Oct 29 22:39:52.562136 ip-10-0-130-252 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 29 22:39:52.562145 ip-10-0-130-252 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 29 22:39:52.562152 ip-10-0-130-252 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 29 22:39:52.562453 ip-10-0-130-252 systemd[1]: Failed to start Kubernetes Kubelet. Oct 29 22:40:02.592539 ip-10-0-130-252 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 29 22:40:02.592554 ip-10-0-130-252 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 311142fcd8c74234830406eb74f93906 -- Oct 29 22:41:44.613966 ip-10-0-130-252 systemd[1]: Starting Kubernetes Kubelet... Oct 29 22:41:45.077045 ip-10-0-130-252 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:41:45.077045 ip-10-0-130-252 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 29 22:41:45.077045 ip-10-0-130-252 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:41:45.077045 ip-10-0-130-252 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:41:45.083185 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083102 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:41:45.083228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083192 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:41:45.083228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083197 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:41:45.083228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083200 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:41:45.083228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083203 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:41:45.083228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083207 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:41:45.083228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083210 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:41:45.083228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083212 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:41:45.083228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083215 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:41:45.083228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083218 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:41:45.083228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083220 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:41:45.083228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083223 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:41:45.083228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083225 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:41:45.083228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083228 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:41:45.083228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083231 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:41:45.083228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083234 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:41:45.083584 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083237 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:41:45.083584 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083239 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:41:45.083584 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083252 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:41:45.083584 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083255 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:41:45.083584 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083257 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:41:45.083584 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083260 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:41:45.083584 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083263 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:41:45.083584 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083265 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:41:45.083584 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083269 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:41:45.083584 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083271 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:41:45.083584 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083273 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:41:45.083584 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083276 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:41:45.083584 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083278 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:41:45.083584 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083281 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:41:45.083584 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083283 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:41:45.083584 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083286 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:41:45.083584 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083289 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:41:45.083584 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083291 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:41:45.083584 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083294 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:41:45.084036 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083296 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:41:45.084036 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083298 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:41:45.084036 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083301 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:41:45.084036 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083303 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:41:45.084036 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083306 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:41:45.084036 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083309 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:41:45.084036 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083311 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:41:45.084036 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083313 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:41:45.084036 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083316 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:41:45.084036 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083318 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:41:45.084036 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083321 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:41:45.084036 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083323 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:41:45.084036 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083325 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:41:45.084036 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083328 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:41:45.084036 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083333 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:41:45.084036 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083337 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:41:45.084036 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083346 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:41:45.084036 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083349 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:41:45.084036 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083352 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:41:45.084036 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083354 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:41:45.084568 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083357 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:41:45.084568 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083359 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:41:45.084568 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083362 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:41:45.084568 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083365 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:41:45.084568 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083367 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:41:45.084568 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083370 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:41:45.084568 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083372 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:41:45.084568 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083375 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:41:45.084568 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083377 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:41:45.084568 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083380 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:41:45.084568 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083382 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:41:45.084568 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083385 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:41:45.084568 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083387 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:41:45.084568 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083390 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:41:45.084568 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083394 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:41:45.084568 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083397 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:41:45.084568 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083400 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:41:45.084568 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083403 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:41:45.084568 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083405 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:41:45.084568 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083408 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:41:45.085323 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083410 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:41:45.085323 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083413 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:41:45.085323 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083415 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:41:45.085323 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083418 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:41:45.085323 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083420 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:41:45.085323 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083423 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:41:45.085323 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083426 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:41:45.085323 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083429 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:41:45.085323 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083431 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:41:45.085323 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083440 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:41:45.085323 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083442 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:41:45.085323 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083445 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:41:45.085323 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083448 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:41:45.085323 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083450 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:41:45.085323 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083453 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:41:45.085323 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083455 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:41:45.085323 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083458 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:41:45.085323 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083460 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:41:45.085323 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083463 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:41:45.085323 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083465 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:41:45.085882 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083468 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:41:45.085882 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083471 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:41:45.085882 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083473 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:41:45.085882 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083476 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:41:45.085882 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083478 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:41:45.085882 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083482 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:41:45.085882 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083484 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:41:45.085882 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083487 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:41:45.085882 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083489 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:41:45.085882 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083492 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:41:45.085882 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.083494 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:41:45.085882 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084798 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:41:45.085882 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084832 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:41:45.085882 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084837 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:41:45.085882 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084841 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:41:45.085882 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084846 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:41:45.085882 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084850 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:41:45.085882 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084854 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:41:45.085882 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084860 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:41:45.085882 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084864 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:41:45.086356 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084868 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:41:45.086356 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084872 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:41:45.086356 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084879 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:41:45.086356 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084883 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:41:45.086356 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084887 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:41:45.086356 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084897 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:41:45.086356 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084901 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:41:45.086356 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084905 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:41:45.086356 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084909 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:41:45.086356 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084915 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:41:45.086356 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084920 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:41:45.086356 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084924 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:41:45.086356 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084928 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:41:45.086356 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084932 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:41:45.086356 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084951 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:41:45.086356 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084956 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:41:45.086356 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084960 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:41:45.086356 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084969 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:41:45.086356 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084973 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:41:45.086835 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084978 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:41:45.086835 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084982 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:41:45.086835 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084986 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:41:45.086835 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084990 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:41:45.086835 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084994 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:41:45.086835 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.084999 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:41:45.086835 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085003 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:41:45.086835 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085007 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:41:45.086835 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085011 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:41:45.086835 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085015 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:41:45.086835 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085019 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:41:45.086835 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085029 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:41:45.086835 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085034 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:41:45.086835 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085038 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:41:45.086835 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085043 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:41:45.086835 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085047 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:41:45.086835 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085052 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:41:45.086835 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085056 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:41:45.086835 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085060 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:41:45.087281 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085065 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:41:45.087281 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085069 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:41:45.087281 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085073 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:41:45.087281 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085077 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:41:45.087281 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085086 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:41:45.087281 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085091 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:41:45.087281 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085095 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:41:45.087281 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085099 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:41:45.087281 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085103 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:41:45.087281 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085107 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:41:45.087281 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085112 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:41:45.087281 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085116 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:41:45.087281 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085120 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:41:45.087281 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085124 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:41:45.087281 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085128 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:41:45.087281 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085132 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:41:45.087281 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085136 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:41:45.087281 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085145 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:41:45.087281 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085149 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:41:45.087281 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085153 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:41:45.087768 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085157 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:41:45.087768 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085161 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:41:45.087768 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085166 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:41:45.087768 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085170 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:41:45.087768 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085175 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:41:45.087768 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085179 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:41:45.087768 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085183 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:41:45.087768 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085187 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:41:45.087768 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085191 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:41:45.087768 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085195 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:41:45.087768 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085205 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:41:45.087768 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085210 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:41:45.087768 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085214 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:41:45.087768 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085218 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:41:45.087768 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085222 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:41:45.087768 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085226 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:41:45.087768 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085233 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:41:45.087768 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085237 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:41:45.087768 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085241 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:41:45.087768 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085246 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:41:45.088255 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085249 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:41:45.088255 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085253 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:41:45.088255 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085262 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:41:45.088255 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085269 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:41:45.088255 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085274 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:41:45.088255 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085279 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:41:45.088255 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085283 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:41:45.088255 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085287 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:41:45.088255 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085294 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:41:45.088255 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085299 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:41:45.088255 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085304 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:41:45.088255 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085308 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:41:45.088255 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085313 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:41:45.088255 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085317 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:41:45.088255 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085327 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:41:45.088255 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085332 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:41:45.088255 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085335 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:41:45.088255 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085339 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:41:45.088255 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.085345 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:41:45.088255 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086092 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:41:45.088734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086101 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:41:45.088734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086105 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:41:45.088734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086108 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:41:45.088734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086110 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:41:45.088734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086113 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:41:45.088734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086115 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:41:45.088734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086118 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:41:45.088734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086121 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:41:45.088734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086123 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:41:45.088734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086126 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:41:45.088734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086129 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:41:45.088734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086131 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:41:45.088734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086134 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:41:45.088734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086137 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:41:45.088734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086140 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:41:45.088734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086142 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:41:45.088734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086145 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:41:45.088734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086147 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:41:45.088734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086150 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:41:45.089187 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086153 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:41:45.089187 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086156 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:41:45.089187 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086159 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:41:45.089187 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086163 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:41:45.089187 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086168 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:41:45.089187 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086172 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:41:45.089187 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086175 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:41:45.089187 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086178 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:41:45.089187 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086181 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:41:45.089187 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086184 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:41:45.089187 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086187 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:41:45.089187 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086190 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:41:45.089187 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086192 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:41:45.089187 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086195 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:41:45.089187 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086198 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:41:45.089187 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086200 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:41:45.089187 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086203 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:41:45.089187 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086205 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:41:45.089187 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086208 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:41:45.089632 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086210 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:41:45.089632 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086213 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:41:45.089632 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086215 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:41:45.089632 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086218 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:41:45.089632 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086220 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:41:45.089632 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086222 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:41:45.089632 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086225 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:41:45.089632 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086228 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:41:45.089632 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086230 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:41:45.089632 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086232 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:41:45.089632 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086235 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:41:45.089632 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086237 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:41:45.089632 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086240 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:41:45.089632 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086242 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:41:45.089632 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086245 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:41:45.089632 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086248 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:41:45.089632 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086251 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:41:45.089632 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086253 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:41:45.089632 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086256 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:41:45.089632 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086258 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:41:45.090121 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086261 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:41:45.090121 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086263 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:41:45.090121 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086266 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:41:45.090121 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086268 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:41:45.090121 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086271 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:41:45.090121 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086273 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:41:45.090121 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086276 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:41:45.090121 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086278 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:41:45.090121 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086281 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:41:45.090121 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086283 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:41:45.090121 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086286 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:41:45.090121 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086289 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:41:45.090121 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086291 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:41:45.090121 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086294 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:41:45.090121 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086296 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:41:45.090121 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086298 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:41:45.090121 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086301 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:41:45.090121 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086304 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:41:45.090121 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086307 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:41:45.090121 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086309 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:41:45.090737 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086312 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:41:45.090737 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086315 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:41:45.090737 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086317 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:41:45.090737 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086320 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:41:45.090737 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086323 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:41:45.090737 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086327 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:41:45.090737 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086331 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:41:45.090737 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086334 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:41:45.090737 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086337 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:41:45.090737 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086340 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:41:45.090737 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086342 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:41:45.090737 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086345 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:41:45.090737 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086347 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:41:45.090737 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086350 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:41:45.090737 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086352 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:41:45.090737 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086354 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:41:45.090737 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086357 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:41:45.090737 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086360 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:41:45.090737 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086362 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:41:45.091178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086365 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:41:45.091178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086368 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:41:45.091178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086370 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:41:45.091178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086374 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:41:45.091178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086377 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:41:45.091178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086380 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:41:45.091178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086382 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:41:45.091178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.086385 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:41:45.091178 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088060 2577 flags.go:64] FLAG: --address="0.0.0.0" Oct 29 22:41:45.091178 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088069 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 29 22:41:45.091178 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088078 2577 flags.go:64] FLAG: --anonymous-auth="true" Oct 29 22:41:45.091178 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088083 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 29 22:41:45.091178 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088087 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 29 22:41:45.091178 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088090 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 29 22:41:45.091178 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088094 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 29 22:41:45.091178 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088099 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 29 22:41:45.091178 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088102 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 29 22:41:45.091178 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088105 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 29 22:41:45.091178 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088108 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 29 22:41:45.091178 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088112 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 29 22:41:45.091178 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088115 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 29 22:41:45.091178 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088118 2577 flags.go:64] FLAG: --cgroup-root="" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088121 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088124 2577 flags.go:64] FLAG: --client-ca-file="" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088126 2577 flags.go:64] FLAG: --cloud-provider="external" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088129 2577 flags.go:64] FLAG: --cluster-dns="[]" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088133 2577 flags.go:64] FLAG: --cluster-domain="" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088136 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088140 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088143 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088146 2577 flags.go:64] FLAG: --container-log-max-files="5" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088151 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088154 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088156 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088160 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088164 2577 flags.go:64] FLAG: --contention-profiling="false" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088167 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088170 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088173 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088176 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088181 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088184 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088187 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088190 2577 flags.go:64] FLAG: --enable-load-reader="false" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088193 2577 flags.go:64] FLAG: --enable-server="true" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088196 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 29 22:41:45.091675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088200 2577 flags.go:64] FLAG: --event-burst="100" Oct 29 22:41:45.092310 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088203 2577 flags.go:64] FLAG: --event-qps="50" Oct 29 22:41:45.092310 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088206 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 29 22:41:45.092310 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088209 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 29 22:41:45.092310 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088212 2577 flags.go:64] FLAG: --eviction-hard="" Oct 29 22:41:45.092310 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088216 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 29 22:41:45.092310 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088219 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 29 22:41:45.092310 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088222 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 29 22:41:45.092310 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088225 2577 flags.go:64] FLAG: --eviction-soft="" Oct 29 22:41:45.092310 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088228 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 29 22:41:45.092310 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088231 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 29 22:41:45.092310 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088234 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 29 22:41:45.092310 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088237 2577 flags.go:64] FLAG: --experimental-mounter-path="" Oct 29 22:41:45.092310 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088240 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 29 22:41:45.092310 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088243 2577 flags.go:64] FLAG: --fail-swap-on="true" Oct 29 22:41:45.092310 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088245 2577 flags.go:64] FLAG: --feature-gates="" Oct 29 22:41:45.092310 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088249 2577 flags.go:64] FLAG: --file-check-frequency="20s" Oct 29 22:41:45.092310 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088252 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 29 22:41:45.092310 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088255 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 29 22:41:45.092310 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088259 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 29 22:41:45.092310 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088261 2577 flags.go:64] FLAG: --healthz-port="10248" Oct 29 22:41:45.092310 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088265 2577 flags.go:64] FLAG: --help="false" Oct 29 22:41:45.092310 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088268 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-130-252.ec2.internal" Oct 29 22:41:45.092310 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088271 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 29 22:41:45.092310 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088274 2577 flags.go:64] FLAG: --http-check-frequency="20s" Oct 29 22:41:45.092902 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088277 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 29 22:41:45.092902 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088281 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 29 22:41:45.092902 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088284 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 29 22:41:45.092902 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088287 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 29 22:41:45.092902 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088290 2577 flags.go:64] FLAG: --image-service-endpoint="" Oct 29 22:41:45.092902 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088293 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 29 22:41:45.092902 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088296 2577 flags.go:64] FLAG: --kube-api-burst="100" Oct 29 22:41:45.092902 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088299 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 29 22:41:45.092902 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088302 2577 flags.go:64] FLAG: --kube-api-qps="50" Oct 29 22:41:45.092902 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088304 2577 flags.go:64] FLAG: --kube-reserved="" Oct 29 22:41:45.092902 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088307 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 29 22:41:45.092902 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088310 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 29 22:41:45.092902 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088313 2577 flags.go:64] FLAG: --kubelet-cgroups="" Oct 29 22:41:45.092902 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088316 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 29 22:41:45.092902 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088319 2577 flags.go:64] FLAG: --lock-file="" Oct 29 22:41:45.092902 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088322 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 29 22:41:45.092902 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088324 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 29 22:41:45.092902 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088327 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 29 22:41:45.092902 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088333 2577 flags.go:64] FLAG: --log-json-split-stream="false" Oct 29 22:41:45.092902 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088336 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 29 22:41:45.092902 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088339 2577 flags.go:64] FLAG: --log-text-split-stream="false" Oct 29 22:41:45.092902 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088342 2577 flags.go:64] FLAG: --logging-format="text" Oct 29 22:41:45.092902 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088345 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088348 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088351 2577 flags.go:64] FLAG: --manifest-url="" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088353 2577 flags.go:64] FLAG: --manifest-url-header="" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088358 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088361 2577 flags.go:64] FLAG: --max-open-files="1000000" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088365 2577 flags.go:64] FLAG: --max-pods="110" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088368 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088371 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088374 2577 flags.go:64] FLAG: --memory-manager-policy="None" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088376 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088379 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088382 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088385 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088393 2577 flags.go:64] FLAG: --node-status-max-images="50" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088396 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088399 2577 flags.go:64] FLAG: --oom-score-adj="-999" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088403 2577 flags.go:64] FLAG: --pod-cidr="" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088405 2577 flags.go:64] FLAG: --pod-manifest-path="" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088408 2577 flags.go:64] FLAG: --pod-max-pids="-1" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088411 2577 flags.go:64] FLAG: --pods-per-core="0" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088414 2577 flags.go:64] FLAG: --port="10250" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088420 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088423 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04f821dbc8a676187" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088426 2577 flags.go:64] FLAG: --qos-reserved="" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088429 2577 flags.go:64] FLAG: --read-only-port="10255" Oct 29 22:41:45.093445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088432 2577 flags.go:64] FLAG: --register-node="true" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088434 2577 flags.go:64] FLAG: --register-with-taints="" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088438 2577 flags.go:64] FLAG: --registry-burst="10" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088441 2577 flags.go:64] FLAG: --registry-qps="5" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088444 2577 flags.go:64] FLAG: --reserved-cpus="" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088447 2577 flags.go:64] FLAG: --reserved-memory="" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088451 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088454 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088457 2577 flags.go:64] FLAG: --rotate-certificates="false" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088460 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088462 2577 flags.go:64] FLAG: --runonce="false" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088465 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088468 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088472 2577 flags.go:64] FLAG: --seccomp-default="false" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088474 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088477 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088480 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088483 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088486 2577 flags.go:64] FLAG: --storage-driver-password="root" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088489 2577 flags.go:64] FLAG: --storage-driver-secure="false" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088492 2577 flags.go:64] FLAG: --storage-driver-table="stats" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088495 2577 flags.go:64] FLAG: --storage-driver-user="root" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088498 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088502 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088505 2577 flags.go:64] FLAG: --system-cgroups="" Oct 29 22:41:45.094096 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088508 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 29 22:41:45.094674 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088513 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 29 22:41:45.094674 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088515 2577 flags.go:64] FLAG: --tls-cert-file="" Oct 29 22:41:45.094674 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088520 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 29 22:41:45.094674 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088523 2577 flags.go:64] FLAG: --tls-min-version="" Oct 29 22:41:45.094674 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088526 2577 flags.go:64] FLAG: --tls-private-key-file="" Oct 29 22:41:45.094674 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088529 2577 flags.go:64] FLAG: --topology-manager-policy="none" Oct 29 22:41:45.094674 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088532 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 29 22:41:45.094674 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088535 2577 flags.go:64] FLAG: --topology-manager-scope="container" Oct 29 22:41:45.094674 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088538 2577 flags.go:64] FLAG: --v="2" Oct 29 22:41:45.094674 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088542 2577 flags.go:64] FLAG: --version="false" Oct 29 22:41:45.094674 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088546 2577 flags.go:64] FLAG: --vmodule="" Oct 29 22:41:45.094674 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088550 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 29 22:41:45.094674 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.088554 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 29 22:41:45.094674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088707 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:41:45.094674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088711 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:41:45.094674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088714 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:41:45.094674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088717 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:41:45.094674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088720 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:41:45.094674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088723 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:41:45.094674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088725 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:41:45.094674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088728 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:41:45.094674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088731 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:41:45.094674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088734 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:41:45.095270 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088736 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:41:45.095270 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088741 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:41:45.095270 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088745 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:41:45.095270 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088749 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:41:45.095270 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088752 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:41:45.095270 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088754 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:41:45.095270 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088757 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:41:45.095270 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088760 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:41:45.095270 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088762 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:41:45.095270 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088765 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:41:45.095270 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088768 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:41:45.095270 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088772 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:41:45.095270 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088774 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:41:45.095270 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088777 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:41:45.095270 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088780 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:41:45.095270 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088782 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:41:45.095270 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088784 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:41:45.095270 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088787 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:41:45.095270 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088789 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:41:45.095734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088792 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:41:45.095734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088794 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:41:45.095734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088797 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:41:45.095734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088801 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:41:45.095734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088804 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:41:45.095734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088807 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:41:45.095734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088810 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:41:45.095734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088814 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:41:45.095734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088816 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:41:45.095734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088819 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:41:45.095734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088822 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:41:45.095734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088824 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:41:45.095734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088827 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:41:45.095734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088829 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:41:45.095734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088832 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:41:45.095734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088834 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:41:45.095734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088837 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:41:45.095734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088839 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:41:45.095734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088842 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:41:45.095734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088844 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:41:45.096211 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088847 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:41:45.096211 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088850 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:41:45.096211 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088852 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:41:45.096211 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088855 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:41:45.096211 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088859 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:41:45.096211 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088861 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:41:45.096211 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088864 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:41:45.096211 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088866 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:41:45.096211 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088869 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:41:45.096211 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088871 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:41:45.096211 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088874 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:41:45.096211 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088876 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:41:45.096211 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088879 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:41:45.096211 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088882 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:41:45.096211 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088884 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:41:45.096211 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088886 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:41:45.096211 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088889 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:41:45.096211 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088892 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:41:45.096211 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088894 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:41:45.096211 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088899 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:41:45.096674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088901 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:41:45.096674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088904 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:41:45.096674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088907 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:41:45.096674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088909 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:41:45.096674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088912 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:41:45.096674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088914 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:41:45.096674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088916 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:41:45.096674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088919 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:41:45.096674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088922 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:41:45.096674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088924 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:41:45.096674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088927 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:41:45.096674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088929 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:41:45.096674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088932 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:41:45.096674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088934 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:41:45.096674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088938 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:41:45.096674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088940 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:41:45.096674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088944 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:41:45.096674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088947 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:41:45.096674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088949 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:41:45.096674 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088952 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:41:45.097176 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088954 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:41:45.097176 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088957 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:41:45.097176 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088959 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:41:45.097176 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088962 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:41:45.097176 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088964 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:41:45.097176 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088967 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:41:45.097176 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088970 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:41:45.097176 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088972 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:41:45.097176 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088975 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:41:45.097176 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088978 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:41:45.097176 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088980 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:41:45.097176 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088984 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:41:45.097176 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088987 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:41:45.097176 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088989 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:41:45.097176 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088991 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:41:45.097176 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088994 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:41:45.097176 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.088996 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:41:45.097579 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.089023 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:41:45.097579 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089058 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:41:45.097579 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089063 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:41:45.097579 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089065 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:41:45.097579 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089068 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:41:45.097579 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089071 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:41:45.097579 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089073 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:41:45.097579 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089076 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:41:45.097579 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089079 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:41:45.097579 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089082 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:41:45.097579 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089085 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:41:45.097579 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089088 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:41:45.097579 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089091 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:41:45.097579 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089094 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:41:45.097579 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089097 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:41:45.097579 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089099 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:41:45.097980 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089102 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:41:45.097980 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089104 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:41:45.097980 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089107 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:41:45.097980 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089109 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:41:45.097980 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089112 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:41:45.097980 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089114 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:41:45.097980 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089117 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:41:45.097980 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089119 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:41:45.097980 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089122 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:41:45.097980 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089124 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:41:45.097980 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089127 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:41:45.097980 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089131 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:41:45.097980 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089133 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:41:45.097980 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089136 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:41:45.097980 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089138 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:41:45.097980 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089141 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:41:45.097980 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089144 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:41:45.097980 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089146 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:41:45.097980 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089149 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:41:45.097980 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089151 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:41:45.098455 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089154 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:41:45.098455 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089156 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:41:45.098455 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089159 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:41:45.098455 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089161 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:41:45.098455 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089164 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:41:45.098455 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089166 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:41:45.098455 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089171 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:41:45.098455 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089176 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:41:45.098455 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089179 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:41:45.098455 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089181 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:41:45.098455 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089184 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:41:45.098455 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089186 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:41:45.098455 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089189 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:41:45.098455 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089192 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:41:45.098455 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089194 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:41:45.098455 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089197 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:41:45.098455 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089200 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:41:45.098455 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089202 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:41:45.098455 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089205 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:41:45.098912 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089207 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:41:45.098912 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089210 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:41:45.098912 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089213 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:41:45.098912 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089215 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:41:45.098912 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089219 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:41:45.098912 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089221 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:41:45.098912 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089224 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:41:45.098912 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089226 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:41:45.098912 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089231 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:41:45.098912 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089234 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:41:45.098912 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089238 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:41:45.098912 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089241 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:41:45.098912 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089243 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:41:45.098912 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089246 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:41:45.098912 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089248 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:41:45.098912 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089250 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:41:45.098912 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089253 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:41:45.098912 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089255 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:41:45.098912 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089258 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:41:45.098912 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089261 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:41:45.099415 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089264 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:41:45.099415 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089267 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:41:45.099415 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089269 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:41:45.099415 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089272 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:41:45.099415 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089274 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:41:45.099415 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089277 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:41:45.099415 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089279 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:41:45.099415 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089282 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:41:45.099415 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089284 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:41:45.099415 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089287 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:41:45.099415 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089289 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:41:45.099415 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089292 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:41:45.099415 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089294 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:41:45.099415 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089297 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:41:45.099415 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089299 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:41:45.099415 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089302 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:41:45.099415 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089307 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:41:45.099415 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089310 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:41:45.099415 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089312 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:41:45.099415 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089315 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:41:45.099916 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089317 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:41:45.099916 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089320 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:41:45.099916 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089323 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:41:45.099916 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089325 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:41:45.099916 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089328 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:41:45.099916 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089330 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:41:45.099916 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089335 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:41:45.099916 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089337 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:41:45.099916 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089339 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:41:45.099916 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089342 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:41:45.099916 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089345 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:41:45.099916 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.089348 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:41:45.099916 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.097216 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 29 22:41:45.099916 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.097233 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 29 22:41:45.099916 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.097244 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 29 22:41:45.099916 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.097248 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 29 22:41:45.099916 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097308 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:41:45.099916 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097313 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:41:45.099916 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097316 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:41:45.099916 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097319 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:41:45.100386 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097321 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:41:45.100386 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097324 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:41:45.100386 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097327 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:41:45.100386 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097330 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:41:45.100386 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097332 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:41:45.100386 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097335 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:41:45.100386 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097337 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:41:45.100386 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097340 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:41:45.100386 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097342 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:41:45.100386 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097345 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:41:45.100386 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097348 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:41:45.100386 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097350 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:41:45.100386 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097353 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:41:45.100386 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097356 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:41:45.100386 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097358 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:41:45.100386 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097360 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:41:45.100386 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097363 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:41:45.100386 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097366 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:41:45.100386 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097368 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:41:45.100852 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097371 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:41:45.100852 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097373 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:41:45.100852 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097376 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:41:45.100852 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097378 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:41:45.100852 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097381 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:41:45.100852 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097384 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:41:45.100852 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097387 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:41:45.100852 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097390 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:41:45.100852 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097393 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:41:45.100852 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097395 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:41:45.100852 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097398 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:41:45.100852 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097400 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:41:45.100852 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097403 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:41:45.100852 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097406 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:41:45.100852 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097408 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:41:45.100852 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097411 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:41:45.100852 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097414 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:41:45.100852 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097417 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:41:45.100852 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097420 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:41:45.101305 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097422 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:41:45.101305 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097425 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:41:45.101305 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097427 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:41:45.101305 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097430 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:41:45.101305 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097433 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:41:45.101305 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097436 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:41:45.101305 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097438 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:41:45.101305 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097441 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:41:45.101305 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097444 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:41:45.101305 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097447 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:41:45.101305 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097449 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:41:45.101305 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097452 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:41:45.101305 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097454 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:41:45.101305 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097457 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:41:45.101305 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097459 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:41:45.101305 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097462 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:41:45.101305 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097465 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:41:45.101305 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097467 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:41:45.101305 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097470 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:41:45.101771 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097473 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:41:45.101771 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097476 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:41:45.101771 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097479 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:41:45.101771 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097482 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:41:45.101771 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097484 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:41:45.101771 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097487 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:41:45.101771 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097489 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:41:45.101771 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097492 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:41:45.101771 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097495 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:41:45.101771 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097498 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:41:45.101771 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097502 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:41:45.101771 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097505 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:41:45.101771 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097507 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:41:45.101771 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097510 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:41:45.101771 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097514 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:41:45.101771 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097517 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:41:45.101771 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097521 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:41:45.101771 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097524 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:41:45.101771 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097527 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:41:45.101771 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097530 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:41:45.102246 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097532 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:41:45.102246 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097535 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:41:45.102246 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097538 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:41:45.102246 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097540 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:41:45.102246 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097543 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:41:45.102246 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097545 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:41:45.102246 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097548 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:41:45.102246 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097551 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:41:45.102246 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097553 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:41:45.102246 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097556 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:41:45.102246 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097558 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:41:45.102246 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097561 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:41:45.102246 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097565 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:41:45.102246 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097567 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:41:45.102246 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097571 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:41:45.102246 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097573 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:41:45.102246 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097576 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:41:45.102246 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097579 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:41:45.102246 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097582 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:41:45.102246 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097585 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:41:45.102899 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097587 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:41:45.102899 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097590 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:41:45.102899 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097592 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:41:45.102899 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097595 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:41:45.102899 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097597 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:41:45.102899 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.097624 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:41:45.102899 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097838 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:41:45.102899 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097843 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:41:45.102899 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097846 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:41:45.102899 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097849 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:41:45.102899 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097851 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:41:45.102899 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097854 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:41:45.102899 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097856 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:41:45.102899 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097859 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:41:45.102899 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097862 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:41:45.102899 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097864 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:41:45.103293 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097868 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:41:45.103293 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097872 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:41:45.103293 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097876 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:41:45.103293 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097879 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:41:45.103293 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097882 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:41:45.103293 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097884 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:41:45.103293 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097888 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:41:45.103293 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097891 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:41:45.103293 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097895 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:41:45.103293 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097898 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:41:45.103293 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097900 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:41:45.103293 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097903 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:41:45.103293 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097906 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:41:45.103293 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097909 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:41:45.103293 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097912 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:41:45.103293 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097914 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:41:45.103293 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097917 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:41:45.103293 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097919 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:41:45.103293 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097922 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:41:45.103752 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097924 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:41:45.103752 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097927 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:41:45.103752 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097929 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:41:45.103752 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097932 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:41:45.103752 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097934 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:41:45.103752 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097937 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:41:45.103752 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097939 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:41:45.103752 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097942 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:41:45.103752 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097944 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:41:45.103752 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097947 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:41:45.103752 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097949 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:41:45.103752 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097952 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:41:45.103752 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097954 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:41:45.103752 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097957 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:41:45.103752 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097959 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:41:45.103752 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097962 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:41:45.103752 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097965 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:41:45.103752 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097967 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:41:45.103752 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097970 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:41:45.103752 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097972 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:41:45.104228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097975 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:41:45.104228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097977 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:41:45.104228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097980 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:41:45.104228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097983 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:41:45.104228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097985 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:41:45.104228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097988 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:41:45.104228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097991 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:41:45.104228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097994 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:41:45.104228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097996 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:41:45.104228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.097999 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:41:45.104228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098002 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:41:45.104228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098004 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:41:45.104228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098007 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:41:45.104228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098009 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:41:45.104228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098011 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:41:45.104228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098014 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:41:45.104228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098016 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:41:45.104228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098019 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:41:45.104228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098021 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:41:45.104228 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098024 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:41:45.104734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098026 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:41:45.104734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098029 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:41:45.104734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098031 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:41:45.104734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098034 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:41:45.104734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098036 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:41:45.104734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098039 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:41:45.104734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098042 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:41:45.104734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098045 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:41:45.104734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098049 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:41:45.104734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098052 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:41:45.104734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098055 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:41:45.104734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098058 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:41:45.104734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098060 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:41:45.104734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098063 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:41:45.104734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098065 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:41:45.104734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098068 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:41:45.104734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098070 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:41:45.104734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098073 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:41:45.104734 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098076 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:41:45.105178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098078 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:41:45.105178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098081 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:41:45.105178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098083 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:41:45.105178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098086 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:41:45.105178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098088 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:41:45.105178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098091 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:41:45.105178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098093 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:41:45.105178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098096 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:41:45.105178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098098 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:41:45.105178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098101 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:41:45.105178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098104 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:41:45.105178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098106 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:41:45.105178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098109 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:41:45.105178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098111 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:41:45.105178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098114 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:41:45.105178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098116 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:41:45.105178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098118 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:41:45.105178 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098121 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:41:45.105706 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.098144 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:41:45.105706 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098185 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:41:45.105706 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098190 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:41:45.105706 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098193 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:41:45.105706 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098196 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:41:45.105706 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098199 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:41:45.105706 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098201 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:41:45.105706 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098204 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:41:45.105706 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098206 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:41:45.105706 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098209 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:41:45.105706 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098211 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:41:45.105706 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098214 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:41:45.105706 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098216 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:41:45.105706 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098218 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:41:45.105706 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098227 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:41:45.105706 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098230 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:41:45.106101 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098232 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:41:45.106101 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098235 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:41:45.106101 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098237 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:41:45.106101 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098240 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:41:45.106101 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098242 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:41:45.106101 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098245 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:41:45.106101 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098247 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:41:45.106101 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098250 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:41:45.106101 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098252 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:41:45.106101 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098255 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:41:45.106101 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098258 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:41:45.106101 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098261 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:41:45.106101 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098265 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:41:45.106101 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098268 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:41:45.106101 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098271 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:41:45.106101 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098274 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:41:45.106101 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098276 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:41:45.106101 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098279 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:41:45.106101 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098281 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:41:45.106541 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098284 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:41:45.106541 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098286 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:41:45.106541 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098289 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:41:45.106541 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098291 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:41:45.106541 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098294 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:41:45.106541 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098296 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:41:45.106541 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098298 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:41:45.106541 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098301 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:41:45.106541 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098303 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:41:45.106541 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098305 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:41:45.106541 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098308 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:41:45.106541 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098310 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:41:45.106541 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098318 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:41:45.106541 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098321 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:41:45.106541 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098324 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:41:45.106541 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098327 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:41:45.106541 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098329 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:41:45.106541 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098331 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:41:45.106541 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098334 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:41:45.107008 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098336 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:41:45.107008 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098339 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:41:45.107008 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098342 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:41:45.107008 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098345 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:41:45.107008 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098347 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:41:45.107008 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098349 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:41:45.107008 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098352 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:41:45.107008 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098354 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:41:45.107008 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098357 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:41:45.107008 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098359 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:41:45.107008 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098362 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:41:45.107008 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098380 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:41:45.107008 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098383 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:41:45.107008 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098386 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:41:45.107008 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098390 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:41:45.107008 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098394 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:41:45.107008 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098396 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:41:45.107008 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098399 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:41:45.107008 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098401 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:41:45.107487 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098404 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:41:45.107487 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098407 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:41:45.107487 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098410 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:41:45.107487 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098412 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:41:45.107487 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098415 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:41:45.107487 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098417 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:41:45.107487 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098420 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:41:45.107487 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098428 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:41:45.107487 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098431 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:41:45.107487 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098437 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:41:45.107487 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098440 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:41:45.107487 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098443 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:41:45.107487 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098445 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:41:45.107487 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098448 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:41:45.107487 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098450 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:41:45.107487 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098453 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:41:45.107487 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098455 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:41:45.107487 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098458 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:41:45.107487 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098461 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:41:45.107487 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098464 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:41:45.108003 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098466 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:41:45.108003 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098469 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:41:45.108003 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098472 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:41:45.108003 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098474 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:41:45.108003 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098478 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:41:45.108003 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098481 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:41:45.108003 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098483 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:41:45.108003 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098486 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:41:45.108003 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098489 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:41:45.108003 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098491 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:41:45.108003 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098494 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:41:45.108003 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098496 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:41:45.108003 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098498 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:41:45.108003 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:45.098501 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:41:45.108003 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.098580 2577 server.go:957] "Client rotation is on, will bootstrap in background" Oct 29 22:41:45.108003 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.101011 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 29 22:41:45.108003 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.102095 2577 server.go:1014] "Starting client certificate rotation" Oct 29 22:41:45.108003 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.102193 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 22:41:45.108003 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.103033 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 22:41:45.108468 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.104363 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:41:45.108468 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.105441 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:41:45.117342 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.117324 2577 log.go:25] "Validated CRI v1 runtime API" Oct 29 22:41:45.121288 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.121273 2577 log.go:25] "Validated CRI v1 image API" Oct 29 22:41:45.122582 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.122555 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 29 22:41:45.125547 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.125524 2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7aac7e64-9e19-4146-b16a-f90e6d54c167:/dev/nvme0n1p3 f8d99165-074d-4f21-86dd-53b3f464fb7c:/dev/nvme0n1p4] Oct 29 22:41:45.125638 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.125545 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 29 22:41:45.129778 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.129621 2577 manager.go:217] Machine: {Timestamp:2026-10-29 22:41:45.129159984 +0000 UTC m=+0.373055131 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3071683 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23000d1d50c1d5a6b50935b5c25ef4 SystemUUID:ec23000d-1d50-c1d5-a6b5-0935b5c25ef4 BootID:311142fc-d8c7-4234-8304-06eb74f93906 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f6:c4:ef:b9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f6:c4:ef:b9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:2c:d6:55:fd:dc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 29 22:41:45.130504 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.130490 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 29 22:41:45.130614 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.130598 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 29 22:41:45.134323 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.134294 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 29 22:41:45.134485 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.134324 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-252.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 29 22:41:45.134567 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.134489 2577 topology_manager.go:143] "Creating topology manager with none policy" Oct 29 22:41:45.134567 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.134502 2577 container_manager_linux.go:311] "Creating device plugin manager" Oct 29 22:41:45.134567 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.134516 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:41:45.134567 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.134533 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:41:45.135830 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.135808 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 29 22:41:45.140533 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.140517 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 29 22:41:45.140664 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.140653 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 29 22:41:45.141668 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.141657 2577 kubelet.go:493] "Attempting to sync node with API server" Oct 29 22:41:45.141742 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.141676 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 29 22:41:45.141742 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.141717 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 29 22:41:45.141742 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.141732 2577 kubelet.go:405] "Adding apiserver pod source" Oct 29 22:41:45.141867 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.141753 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 29 22:41:45.142113 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.142099 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:41:45.144553 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.144533 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 29 22:41:45.151088 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.151073 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:41:45.151947 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.151934 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 29 22:41:45.152000 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.151965 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 29 22:41:45.152119 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.152109 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 29 22:41:45.152151 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.152129 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 29 22:41:45.152151 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.152136 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 29 22:41:45.152151 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.152143 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 29 22:41:45.152151 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.152149 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 29 22:41:45.152255 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.152156 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 29 22:41:45.152848 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.152840 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 29 22:41:45.152884 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.152849 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 29 22:41:45.152884 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.152856 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 29 22:41:45.152884 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.152862 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 29 22:41:45.152884 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.152878 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 29 22:41:45.153006 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.152888 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 29 22:41:45.153761 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.153752 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 29 22:41:45.153799 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.153762 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 29 22:41:45.157732 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.157718 2577 server.go:1263] "Started kubelet" Oct 29 22:41:45.157882 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.157830 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 29 22:41:45.158029 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.158008 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 29 22:41:45.158126 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.157883 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 29 22:41:45.158462 ip-10-0-130-252 systemd[1]: Started Kubernetes Kubelet. Oct 29 22:41:45.162753 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.162731 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 29 22:41:45.164315 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.164291 2577 server.go:317] "Adding debug handlers to kubelet server" Oct 29 22:41:45.167887 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:45.167868 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 29 22:41:45.170392 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.170376 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 29 22:41:45.170461 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.170381 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 29 22:41:45.170514 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:45.170493 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-252.ec2.internal\" not found" Oct 29 22:41:45.171182 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.171162 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 29 22:41:45.171182 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.171185 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 29 22:41:45.171304 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.171235 2577 reconstruct.go:97] "Volume reconstruction finished" Oct 29 22:41:45.171304 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.171244 2577 reconciler.go:29] "Reconciler: start to sync state" Oct 29 22:41:45.171483 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.171161 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 29 22:41:45.176048 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.176027 2577 factory.go:55] Registering systemd factory Oct 29 22:41:45.176129 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.176052 2577 factory.go:223] Registration of the systemd container factory successfully Oct 29 22:41:45.176279 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.176267 2577 factory.go:153] Registering CRI-O factory Oct 29 22:41:45.176350 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.176283 2577 factory.go:223] Registration of the crio container factory successfully Oct 29 22:41:45.176350 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.176333 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 29 22:41:45.176448 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.176357 2577 factory.go:103] Registering Raw factory Oct 29 22:41:45.176448 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.176374 2577 manager.go:1199] Started watching for new ooms in manager Oct 29 22:41:45.177513 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.177484 2577 manager.go:319] Starting recovery of all containers Oct 29 22:41:45.187782 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.187767 2577 manager.go:324] Recovery completed Oct 29 22:41:45.191182 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.191162 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-252.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 29 22:41:45.191279 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:45.191178 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-252.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 29 22:41:45.191929 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.191916 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:41:45.192510 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:45.191334 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-252.ec2.internal.18e321503ec29825 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-252.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-252.ec2.internal,},FirstTimestamp:2026-10-29 22:41:45.157629989 +0000 UTC m=+0.401525132,LastTimestamp:2026-10-29 22:41:45.157629989 +0000 UTC m=+0.401525132,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-252.ec2.internal,}" Oct 29 22:41:45.193530 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.193516 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:41:45.193604 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.193543 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:41:45.193604 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.193553 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:41:45.194107 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.194085 2577 cpu_manager.go:226] "Starting" policy="none" Oct 29 22:41:45.194107 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.194096 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 29 22:41:45.194193 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.194111 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 29 22:41:45.194193 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.194139 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:41:45.194859 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.194844 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:41:45.194899 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.194872 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:41:45.194899 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.194882 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:41:45.196373 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.196361 2577 policy_none.go:50] "Start" Oct 29 22:41:45.196411 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.196377 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 29 22:41:45.196411 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.196387 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 29 22:41:45.198343 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.198333 2577 policy_none.go:44] "Start" Oct 29 22:41:45.205193 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:45.205063 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 29 22:41:45.205274 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:45.205074 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 29 22:41:45.205274 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:45.205119 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-252.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 29 22:41:45.211126 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.211102 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 29 22:41:45.217867 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.217848 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xnq8m" Oct 29 22:41:45.219798 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:45.219727 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-252.ec2.internal.18e3215040e664c5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-252.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-252.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-252.ec2.internal,},FirstTimestamp:2026-10-29 22:41:45.193530565 +0000 UTC m=+0.437425710,LastTimestamp:2026-10-29 22:41:45.193530565 +0000 UTC m=+0.437425710,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-252.ec2.internal,}" Oct 29 22:41:45.253843 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.229509 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xnq8m" Oct 29 22:41:45.253843 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.236250 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 29 22:41:45.253843 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.240438 2577 manager.go:341] "Starting Device Plugin manager" Oct 29 22:41:45.253843 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:45.241098 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 29 22:41:45.253843 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.241109 2577 server.go:87] "Starting device plugin registration server" Oct 29 22:41:45.253843 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.241338 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 29 22:41:45.253843 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.241348 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 29 22:41:45.253843 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.241475 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 29 22:41:45.253843 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.241548 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 29 22:41:45.253843 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.241556 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 29 22:41:45.253843 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:45.242058 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 29 22:41:45.253843 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:45.242095 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-252.ec2.internal\" not found" Oct 29 22:41:45.253843 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.245113 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 29 22:41:45.253843 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.246354 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 29 22:41:45.253843 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.246374 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 29 22:41:45.253843 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.246395 2577 kubelet.go:2521] "Starting kubelet main sync loop" Oct 29 22:41:45.253843 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:45.246434 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 29 22:41:45.268614 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.268595 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:41:45.342389 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.342328 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:41:45.343266 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.343239 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:41:45.343394 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.343282 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:41:45.343394 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.343296 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:41:45.343394 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.343325 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-252.ec2.internal" Oct 29 22:41:45.347423 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.347405 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-252.ec2.internal"] Oct 29 22:41:45.347475 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.347462 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:41:45.348223 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.348200 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:41:45.348311 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.348232 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:41:45.348311 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.348248 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:41:45.350432 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.350420 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:41:45.350588 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.350574 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" Oct 29 22:41:45.350626 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.350604 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:41:45.351125 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.351110 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:41:45.351195 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.351131 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:41:45.351195 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.351141 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:41:45.351195 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.351154 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:41:45.351195 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.351166 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:41:45.351320 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.351154 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:41:45.353209 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.353195 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-252.ec2.internal" Oct 29 22:41:45.353283 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.353219 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:41:45.353873 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.353858 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:41:45.353933 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.353888 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:41:45.353933 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.353905 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:41:45.363059 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.363043 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-252.ec2.internal" Oct 29 22:41:45.363154 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:45.363068 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-252.ec2.internal\": node \"ip-10-0-130-252.ec2.internal\" not found" Oct 29 22:41:45.382111 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:45.382090 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-252.ec2.internal\" not found" node="ip-10-0-130-252.ec2.internal" Oct 29 22:41:45.384326 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:45.384310 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-252.ec2.internal\" not found" node="ip-10-0-130-252.ec2.internal" Oct 29 22:41:45.408080 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:45.408054 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-252.ec2.internal\" not found" Oct 29 22:41:45.472972 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.472941 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/83fee9c1d2770428373c9940b59a21be-config\") pod \"kube-apiserver-proxy-ip-10-0-130-252.ec2.internal\" (UID: \"83fee9c1d2770428373c9940b59a21be\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-252.ec2.internal" Oct 29 22:41:45.472972 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.472972 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f8a08c2f1d6fb4fc298c3bba521bafd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal\" (UID: \"3f8a08c2f1d6fb4fc298c3bba521bafd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" Oct 29 22:41:45.473165 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.472990 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f8a08c2f1d6fb4fc298c3bba521bafd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal\" (UID: \"3f8a08c2f1d6fb4fc298c3bba521bafd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" Oct 29 22:41:45.508913 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:45.508884 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-252.ec2.internal\" not found" Oct 29 22:41:45.573478 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.573454 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f8a08c2f1d6fb4fc298c3bba521bafd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal\" (UID: \"3f8a08c2f1d6fb4fc298c3bba521bafd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" Oct 29 22:41:45.573555 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.573482 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/83fee9c1d2770428373c9940b59a21be-config\") pod \"kube-apiserver-proxy-ip-10-0-130-252.ec2.internal\" (UID: \"83fee9c1d2770428373c9940b59a21be\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-252.ec2.internal" Oct 29 22:41:45.573555 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.573497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f8a08c2f1d6fb4fc298c3bba521bafd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal\" (UID: \"3f8a08c2f1d6fb4fc298c3bba521bafd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" Oct 29 22:41:45.573555 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.573552 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f8a08c2f1d6fb4fc298c3bba521bafd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal\" (UID: \"3f8a08c2f1d6fb4fc298c3bba521bafd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" Oct 29 22:41:45.573677 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.573558 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f8a08c2f1d6fb4fc298c3bba521bafd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal\" (UID: \"3f8a08c2f1d6fb4fc298c3bba521bafd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" Oct 29 22:41:45.573677 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.573582 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/83fee9c1d2770428373c9940b59a21be-config\") pod \"kube-apiserver-proxy-ip-10-0-130-252.ec2.internal\" (UID: \"83fee9c1d2770428373c9940b59a21be\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-252.ec2.internal" Oct 29 22:41:45.609275 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:45.609221 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-252.ec2.internal\" not found" Oct 29 22:41:45.686871 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.686845 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" Oct 29 22:41:45.689945 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:45.689929 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-252.ec2.internal" Oct 29 22:41:45.709324 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:45.709298 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-252.ec2.internal\" not found" Oct 29 22:41:45.809789 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:45.809743 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-252.ec2.internal\" not found" Oct 29 22:41:45.910315 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:45.910246 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-252.ec2.internal\" not found" Oct 29 22:41:46.010781 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:46.010753 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-252.ec2.internal\" not found" Oct 29 22:41:46.102295 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:46.102268 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 29 22:41:46.102761 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:46.102415 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 29 22:41:46.111656 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:46.111568 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-252.ec2.internal\" not found" Oct 29 22:41:46.184002 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:46.183931 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:41:46.212648 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:46.212609 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-252.ec2.internal\" not found" Oct 29 22:41:46.227080 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:46.227054 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8a08c2f1d6fb4fc298c3bba521bafd.slice/crio-43eb1690a2264835c978f4dcc58bf4cfc8c17d25c0ff29a4f73e3f8922099f95 WatchSource:0}: Error finding container 43eb1690a2264835c978f4dcc58bf4cfc8c17d25c0ff29a4f73e3f8922099f95: Status 404 returned error can't find the container with id 43eb1690a2264835c978f4dcc58bf4cfc8c17d25c0ff29a4f73e3f8922099f95 Oct 29 22:41:46.233135 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:46.233120 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:41:46.236521 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:46.236494 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:36:46 +0000 UTC" deadline="2028-07-23 14:47:17.537903718 +0000 UTC" Oct 29 22:41:46.236521 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:46.236520 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15184h5m31.301386776s" Oct 29 22:41:46.249869 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:46.249832 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" event={"ID":"3f8a08c2f1d6fb4fc298c3bba521bafd","Type":"ContainerStarted","Data":"43eb1690a2264835c978f4dcc58bf4cfc8c17d25c0ff29a4f73e3f8922099f95"} Oct 29 22:41:46.264841 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:46.264820 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83fee9c1d2770428373c9940b59a21be.slice/crio-16b261e674b5fc0221acefaf8aa0ed6b1b98af78187d75b09e1006787f60a040 WatchSource:0}: Error finding container 16b261e674b5fc0221acefaf8aa0ed6b1b98af78187d75b09e1006787f60a040: Status 404 returned error can't find the container with id 16b261e674b5fc0221acefaf8aa0ed6b1b98af78187d75b09e1006787f60a040 Oct 29 22:41:46.268145 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:46.268130 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:41:46.269142 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:46.269125 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:41:46.269223 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:46.269154 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:41:46.269223 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:46.269164 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:41:46.313390 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:46.313365 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-252.ec2.internal\" not found" Oct 29 22:41:46.412593 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:46.412563 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:41:46.413521 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:46.413505 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-252.ec2.internal\" not found" Oct 29 22:41:46.514053 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:46.513981 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-252.ec2.internal\" not found" Oct 29 22:41:46.615086 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:46.615053 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-252.ec2.internal\" not found" Oct 29 22:41:48.027370 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:48.027321 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-252.ec2.internal\" not found" Oct 29 22:41:48.075381 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.075349 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:41:48.082449 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.082380 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" Oct 29 22:41:48.107304 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.107271 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:41:48.108484 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.108462 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-252.ec2.internal" Oct 29 22:41:48.126994 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.126969 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:41:48.455057 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.454981 2577 apiserver.go:52] "Watching apiserver" Oct 29 22:41:48.482667 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.482635 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 29 22:41:48.496115 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.496089 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 29 22:41:48.496535 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.496511 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-bzvh4","openshift-network-diagnostics/network-check-target-mbv7x","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal","openshift-multus/network-metrics-daemon-29dzf","openshift-image-registry/node-ca-7kgv6","openshift-dns/node-resolver-cw2sv","kube-system/global-pull-secret-syncer-ldgnp","openshift-monitoring/node-exporter-shnd6","openshift-ovn-kubernetes/ovnkube-node-9b8jf","openshift-multus/multus-vdz2c","openshift-multus/multus-additional-cni-plugins-rdtdb","kube-system/konnectivity-agent-jzjsw","openshift-cluster-node-tuning-operator/tuned-m4s7g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8","kube-system/kube-apiserver-proxy-ip-10-0-130-252.ec2.internal"] Oct 29 22:41:48.499405 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.499383 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.501875 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.501853 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:41:48.504161 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.504144 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:41:48.506462 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.506239 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:41:48.506462 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:48.506306 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dzf" podUID="a8ad4438-da5a-4f3a-bb5b-07c330aef792" Oct 29 22:41:48.509264 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.508719 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7kgv6" Oct 29 22:41:48.510701 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.510680 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cw2sv" Oct 29 22:41:48.512844 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.512828 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.514546 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.514522 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 29 22:41:48.514637 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.514611 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 29 22:41:48.514688 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.514614 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 29 22:41:48.514808 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.514791 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-wccb2\"" Oct 29 22:41:48.514863 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.514526 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 29 22:41:48.515316 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.515297 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.516266 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.516226 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 29 22:41:48.516448 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.516434 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 29 22:41:48.516514 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.516504 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vd45d\"" Oct 29 22:41:48.518627 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.518576 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jzjsw" Oct 29 22:41:48.518627 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.518603 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 29 22:41:48.518820 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.518773 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 29 22:41:48.519021 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.518990 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 29 22:41:48.519274 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.519242 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rfj6j\"" Oct 29 22:41:48.521368 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.521038 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bzvh4" Oct 29 22:41:48.524153 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.524135 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.529415 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.529396 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:41:48.529512 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:48.529468 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldgnp" podUID="fd35650d-2c53-4230-8518-c4cb05b59558" Oct 29 22:41:48.530460 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.530432 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 29 22:41:48.530541 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.530496 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xh74b" Oct 29 22:41:48.532193 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.532175 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:41:48.532301 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:48.532252 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbv7x" podUID="b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff" Oct 29 22:41:48.542562 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.542252 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xh74b" Oct 29 22:41:48.563957 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.563931 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-252.ec2.internal" event={"ID":"83fee9c1d2770428373c9940b59a21be","Type":"ContainerStarted","Data":"16b261e674b5fc0221acefaf8aa0ed6b1b98af78187d75b09e1006787f60a040"} Oct 29 22:41:48.584136 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.584110 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 29 22:41:48.594826 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.594800 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ea8559a-aa64-4d00-84c0-2c9405e32606-ovnkube-config\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.594931 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.594838 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/99a7b227-aad8-414f-8668-70a5660d6a6e-iptables-alerter-script\") pod \"iptables-alerter-bzvh4\" (UID: \"99a7b227-aad8-414f-8668-70a5660d6a6e\") " pod="openshift-network-operator/iptables-alerter-bzvh4" Oct 29 22:41:48.594931 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.594862 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb8pj\" (UniqueName: \"kubernetes.io/projected/c147a7be-d6f6-458e-8b0d-70da7a3c7026-kube-api-access-qb8pj\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:41:48.594931 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.594887 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-sys\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.594931 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.594911 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-multus-cni-dir\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.595135 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.594933 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-var-lib-cni-bin\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.595135 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.594955 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-hostroot\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.595135 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.594980 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6ll2\" (UniqueName: \"kubernetes.io/projected/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff-kube-api-access-f6ll2\") pod \"network-check-target-mbv7x\" (UID: \"b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff\") " pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:41:48.595135 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595005 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4845\" (UniqueName: \"kubernetes.io/projected/a8ad4438-da5a-4f3a-bb5b-07c330aef792-kube-api-access-x4845\") pod \"network-metrics-daemon-29dzf\" (UID: \"a8ad4438-da5a-4f3a-bb5b-07c330aef792\") " pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:41:48.595135 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595028 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-sysconfig\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.595135 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595050 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-kubernetes\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.595135 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595072 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-systemd\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.595135 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595094 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-cni-bin\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.595135 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595118 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-registration-dir\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:41:48.595644 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595141 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-slash\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.595644 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595164 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-run-openvswitch\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.595644 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595189 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45m2j\" (UniqueName: \"kubernetes.io/projected/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-kube-api-access-45m2j\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.595644 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595233 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4916614f-8a34-491d-b4ad-bae72c14b140-root\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.595644 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595258 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-sys-fs\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:41:48.595644 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595282 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klzkm\" (UniqueName: \"kubernetes.io/projected/6ea8559a-aa64-4d00-84c0-2c9405e32606-kube-api-access-klzkm\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.595644 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595372 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-textfile\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.595644 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595468 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f99954e5-6dfe-457f-9a8f-22752798bb5d-serviceca\") pod \"node-ca-7kgv6\" (UID: \"f99954e5-6dfe-457f-9a8f-22752798bb5d\") " pod="openshift-image-registry/node-ca-7kgv6" Oct 29 22:41:48.595644 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595524 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-sysctl-d\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.595644 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595564 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-run\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.595644 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595603 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-system-cni-dir\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.595644 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595626 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-multus-socket-dir-parent\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.595644 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595646 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-run-netns\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.596123 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595667 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-etc-selinux\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:41:48.596123 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595710 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f99954e5-6dfe-457f-9a8f-22752798bb5d-host\") pod \"node-ca-7kgv6\" (UID: \"f99954e5-6dfe-457f-9a8f-22752798bb5d\") " pod="openshift-image-registry/node-ca-7kgv6" Oct 29 22:41:48.596123 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595741 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ea8559a-aa64-4d00-84c0-2c9405e32606-env-overrides\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.596123 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595815 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c147a7be-d6f6-458e-8b0d-70da7a3c7026-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:41:48.596123 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595878 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-socket-dir\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:41:48.596123 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595917 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-run-systemd\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.596123 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595952 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c147a7be-d6f6-458e-8b0d-70da7a3c7026-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:41:48.596123 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.595980 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-tmp\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.596123 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596026 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4916614f-8a34-491d-b4ad-bae72c14b140-sys\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.596123 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596063 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w9d9\" (UniqueName: \"kubernetes.io/projected/af139c4a-79a2-47b4-95e4-493e3430be98-kube-api-access-9w9d9\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:41:48.596123 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596104 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvp94\" (UniqueName: \"kubernetes.io/projected/99a7b227-aad8-414f-8668-70a5660d6a6e-kube-api-access-qvp94\") pod \"iptables-alerter-bzvh4\" (UID: \"99a7b227-aad8-414f-8668-70a5660d6a6e\") " pod="openshift-network-operator/iptables-alerter-bzvh4" Oct 29 22:41:48.596633 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596146 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4916614f-8a34-491d-b4ad-bae72c14b140-metrics-client-ca\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.596633 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596167 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-kubelet\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.596633 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596234 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-node-log\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.596633 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596277 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-cni-netd\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.596633 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596308 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c147a7be-d6f6-458e-8b0d-70da7a3c7026-cnibin\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:41:48.596633 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596334 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8ad4438-da5a-4f3a-bb5b-07c330aef792-metrics-certs\") pod \"network-metrics-daemon-29dzf\" (UID: \"a8ad4438-da5a-4f3a-bb5b-07c330aef792\") " pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:41:48.596633 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596351 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33360cd4-4738-4c3d-bc9f-9f5003408589-hosts-file\") pod \"node-resolver-cw2sv\" (UID: \"33360cd4-4738-4c3d-bc9f-9f5003408589\") " pod="openshift-dns/node-resolver-cw2sv" Oct 29 22:41:48.596633 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596386 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fd35650d-2c53-4230-8518-c4cb05b59558-kubelet-config\") pod \"global-pull-secret-syncer-ldgnp\" (UID: \"fd35650d-2c53-4230-8518-c4cb05b59558\") " pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:41:48.596633 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596421 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fd35650d-2c53-4230-8518-c4cb05b59558-dbus\") pod \"global-pull-secret-syncer-ldgnp\" (UID: \"fd35650d-2c53-4230-8518-c4cb05b59558\") " pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:41:48.596633 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596457 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-run-ovn-kubernetes\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.596633 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596496 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c147a7be-d6f6-458e-8b0d-70da7a3c7026-system-cni-dir\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:41:48.596633 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596539 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-run-k8s-cni-cncf-io\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.596633 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596577 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-var-lib-cni-multus\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.596633 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596618 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-multus-conf-dir\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.597263 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596647 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0061f2cf-c501-47b0-a89b-3e244f9c313d-agent-certs\") pod \"konnectivity-agent-jzjsw\" (UID: \"0061f2cf-c501-47b0-a89b-3e244f9c313d\") " pod="kube-system/konnectivity-agent-jzjsw" Oct 29 22:41:48.597263 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596673 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c147a7be-d6f6-458e-8b0d-70da7a3c7026-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:41:48.597263 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596730 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-sysctl-conf\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.597263 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596761 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-device-dir\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:41:48.597263 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596782 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsddk\" (UniqueName: \"kubernetes.io/projected/f99954e5-6dfe-457f-9a8f-22752798bb5d-kube-api-access-hsddk\") pod \"node-ca-7kgv6\" (UID: \"f99954e5-6dfe-457f-9a8f-22752798bb5d\") " pod="openshift-image-registry/node-ca-7kgv6" Oct 29 22:41:48.597263 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596811 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-run-netns\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.597263 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596855 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ea8559a-aa64-4d00-84c0-2c9405e32606-ovn-node-metrics-cert\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.597263 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596893 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-wtmp\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.597263 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596920 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-etc-openvswitch\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.597263 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596945 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33360cd4-4738-4c3d-bc9f-9f5003408589-tmp-dir\") pod \"node-resolver-cw2sv\" (UID: \"33360cd4-4738-4c3d-bc9f-9f5003408589\") " pod="openshift-dns/node-resolver-cw2sv" Oct 29 22:41:48.597263 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.596983 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd35650d-2c53-4230-8518-c4cb05b59558-original-pull-secret\") pod \"global-pull-secret-syncer-ldgnp\" (UID: \"fd35650d-2c53-4230-8518-c4cb05b59558\") " pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:41:48.597263 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597019 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51fcfffe-f89d-4064-a0e3-45461cc62570-cni-binary-copy\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.597263 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597046 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-var-lib-kubelet\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.597263 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597067 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51fcfffe-f89d-4064-a0e3-45461cc62570-multus-daemon-config\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.597263 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597090 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-systemd-units\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.597263 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597117 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0061f2cf-c501-47b0-a89b-3e244f9c313d-konnectivity-ca\") pod \"konnectivity-agent-jzjsw\" (UID: \"0061f2cf-c501-47b0-a89b-3e244f9c313d\") " pod="kube-system/konnectivity-agent-jzjsw" Oct 29 22:41:48.597976 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597145 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99a7b227-aad8-414f-8668-70a5660d6a6e-host-slash\") pod \"iptables-alerter-bzvh4\" (UID: \"99a7b227-aad8-414f-8668-70a5660d6a6e\") " pod="openshift-network-operator/iptables-alerter-bzvh4" Oct 29 22:41:48.597976 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597169 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-var-lib-kubelet\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.597976 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597226 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd8z9\" (UniqueName: \"kubernetes.io/projected/51fcfffe-f89d-4064-a0e3-45461cc62570-kube-api-access-wd8z9\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.597976 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597253 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:41:48.597976 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597273 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-var-lib-openvswitch\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.597976 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597296 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-log-socket\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.597976 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597314 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.597976 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597346 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ea8559a-aa64-4d00-84c0-2c9405e32606-ovnkube-script-lib\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.597976 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597370 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmn55\" (UniqueName: \"kubernetes.io/projected/33360cd4-4738-4c3d-bc9f-9f5003408589-kube-api-access-fmn55\") pod \"node-resolver-cw2sv\" (UID: \"33360cd4-4738-4c3d-bc9f-9f5003408589\") " pod="openshift-dns/node-resolver-cw2sv" Oct 29 22:41:48.597976 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597390 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-host\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.597976 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597404 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-run-ovn\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.597976 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597418 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c147a7be-d6f6-458e-8b0d-70da7a3c7026-cni-binary-copy\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:41:48.597976 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597438 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-modprobe-d\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.597976 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597471 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-tuned\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.597976 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597490 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-tls\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.597976 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597507 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.598689 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597532 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-accelerators-collector-config\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.598689 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597558 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d5dd\" (UniqueName: \"kubernetes.io/projected/4916614f-8a34-491d-b4ad-bae72c14b140-kube-api-access-6d5dd\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.598689 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597622 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-lib-modules\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.598689 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597643 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-cnibin\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.598689 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597667 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-run-multus-certs\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.598689 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.597685 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-etc-kubernetes\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.615187 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.615116 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 29 22:41:48.615344 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.615190 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 29 22:41:48.615514 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.615495 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 29 22:41:48.615514 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.615513 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 29 22:41:48.615681 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.615530 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 29 22:41:48.615776 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.615759 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 29 22:41:48.615838 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.615801 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f9xcm\"" Oct 29 22:41:48.615895 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.615832 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 29 22:41:48.615895 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.615863 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 29 22:41:48.615895 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.615871 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mhfn7\"" Oct 29 22:41:48.615895 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.615838 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 29 22:41:48.616080 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.615890 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:41:48.616080 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.615923 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:41:48.616080 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.615967 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 29 22:41:48.627860 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.627841 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 29 22:41:48.627860 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.627854 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g9cd8\"" Oct 29 22:41:48.628020 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.627879 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 29 22:41:48.628020 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.627846 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 29 22:41:48.628020 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.627897 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 29 22:41:48.628020 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.627907 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p6rt5\"" Oct 29 22:41:48.628020 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.627926 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 29 22:41:48.628020 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.627939 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kr4qv\"" Oct 29 22:41:48.628020 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.627975 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pd9xt\"" Oct 29 22:41:48.628020 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.627996 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 29 22:41:48.628020 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.627996 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 29 22:41:48.628020 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.627934 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 29 22:41:48.628020 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.628014 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8l2jp\"" Oct 29 22:41:48.628020 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.628026 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 29 22:41:48.628568 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.628021 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 29 22:41:48.628568 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.628235 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 29 22:41:48.628568 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.628288 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 29 22:41:48.653575 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.653552 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:41:48.698644 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.698619 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-etc-selinux\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:41:48.698644 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.698649 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f99954e5-6dfe-457f-9a8f-22752798bb5d-host\") pod \"node-ca-7kgv6\" (UID: \"f99954e5-6dfe-457f-9a8f-22752798bb5d\") " pod="openshift-image-registry/node-ca-7kgv6" Oct 29 22:41:48.698857 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.698664 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ea8559a-aa64-4d00-84c0-2c9405e32606-env-overrides\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.698857 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.698692 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c147a7be-d6f6-458e-8b0d-70da7a3c7026-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:41:48.698857 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.698716 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-socket-dir\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:41:48.698857 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.698738 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-run-systemd\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.698857 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.698747 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f99954e5-6dfe-457f-9a8f-22752798bb5d-host\") pod \"node-ca-7kgv6\" (UID: \"f99954e5-6dfe-457f-9a8f-22752798bb5d\") " pod="openshift-image-registry/node-ca-7kgv6" Oct 29 22:41:48.698857 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.698762 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c147a7be-d6f6-458e-8b0d-70da7a3c7026-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:41:48.698857 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.698787 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-tmp\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.698857 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.698801 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-run-systemd\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.698857 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.698806 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-etc-selinux\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:41:48.698857 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.698860 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4916614f-8a34-491d-b4ad-bae72c14b140-sys\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.699221 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.698867 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-socket-dir\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:41:48.699221 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.698810 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4916614f-8a34-491d-b4ad-bae72c14b140-sys\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.699221 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.698929 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w9d9\" (UniqueName: \"kubernetes.io/projected/af139c4a-79a2-47b4-95e4-493e3430be98-kube-api-access-9w9d9\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:41:48.699221 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.698958 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvp94\" (UniqueName: \"kubernetes.io/projected/99a7b227-aad8-414f-8668-70a5660d6a6e-kube-api-access-qvp94\") pod \"iptables-alerter-bzvh4\" (UID: \"99a7b227-aad8-414f-8668-70a5660d6a6e\") " pod="openshift-network-operator/iptables-alerter-bzvh4" Oct 29 22:41:48.699221 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.698981 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4916614f-8a34-491d-b4ad-bae72c14b140-metrics-client-ca\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.699221 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-kubelet\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.699221 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699029 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-node-log\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.699221 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699052 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-cni-netd\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.699221 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699074 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c147a7be-d6f6-458e-8b0d-70da7a3c7026-cnibin\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:41:48.699221 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699097 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8ad4438-da5a-4f3a-bb5b-07c330aef792-metrics-certs\") pod \"network-metrics-daemon-29dzf\" (UID: \"a8ad4438-da5a-4f3a-bb5b-07c330aef792\") " pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:41:48.699221 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699119 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33360cd4-4738-4c3d-bc9f-9f5003408589-hosts-file\") pod \"node-resolver-cw2sv\" (UID: \"33360cd4-4738-4c3d-bc9f-9f5003408589\") " pod="openshift-dns/node-resolver-cw2sv" Oct 29 22:41:48.699221 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699147 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fd35650d-2c53-4230-8518-c4cb05b59558-kubelet-config\") pod \"global-pull-secret-syncer-ldgnp\" (UID: \"fd35650d-2c53-4230-8518-c4cb05b59558\") " pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:41:48.699221 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699169 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-kubelet\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.699865 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699231 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ea8559a-aa64-4d00-84c0-2c9405e32606-env-overrides\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.699865 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699246 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-node-log\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.699865 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699171 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fd35650d-2c53-4230-8518-c4cb05b59558-dbus\") pod \"global-pull-secret-syncer-ldgnp\" (UID: \"fd35650d-2c53-4230-8518-c4cb05b59558\") " pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:41:48.699865 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699234 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c147a7be-d6f6-458e-8b0d-70da7a3c7026-cnibin\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:41:48.699865 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699099 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 29 22:41:48.699865 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699287 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-run-ovn-kubernetes\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.699865 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699318 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c147a7be-d6f6-458e-8b0d-70da7a3c7026-system-cni-dir\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:41:48.699865 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699350 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-run-k8s-cni-cncf-io\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.699865 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699373 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-var-lib-cni-multus\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.699865 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699395 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-multus-conf-dir\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.699865 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699406 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c147a7be-d6f6-458e-8b0d-70da7a3c7026-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:41:48.699865 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699419 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0061f2cf-c501-47b0-a89b-3e244f9c313d-agent-certs\") pod \"konnectivity-agent-jzjsw\" (UID: \"0061f2cf-c501-47b0-a89b-3e244f9c313d\") " pod="kube-system/konnectivity-agent-jzjsw" Oct 29 22:41:48.699865 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699461 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-cni-netd\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.699865 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699462 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fd35650d-2c53-4230-8518-c4cb05b59558-dbus\") pod \"global-pull-secret-syncer-ldgnp\" (UID: \"fd35650d-2c53-4230-8518-c4cb05b59558\") " pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:41:48.699865 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699499 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c147a7be-d6f6-458e-8b0d-70da7a3c7026-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:41:48.699865 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699507 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fd35650d-2c53-4230-8518-c4cb05b59558-kubelet-config\") pod \"global-pull-secret-syncer-ldgnp\" (UID: \"fd35650d-2c53-4230-8518-c4cb05b59558\") " pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:41:48.699865 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699522 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-sysctl-conf\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.699865 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699547 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-device-dir\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:41:48.700648 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699569 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsddk\" (UniqueName: \"kubernetes.io/projected/f99954e5-6dfe-457f-9a8f-22752798bb5d-kube-api-access-hsddk\") pod \"node-ca-7kgv6\" (UID: \"f99954e5-6dfe-457f-9a8f-22752798bb5d\") " pod="openshift-image-registry/node-ca-7kgv6" Oct 29 22:41:48.700648 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699593 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-run-netns\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.700648 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699620 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ea8559a-aa64-4d00-84c0-2c9405e32606-ovn-node-metrics-cert\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.700648 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699644 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-wtmp\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.700648 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699667 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-etc-openvswitch\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.700648 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699690 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33360cd4-4738-4c3d-bc9f-9f5003408589-tmp-dir\") pod \"node-resolver-cw2sv\" (UID: \"33360cd4-4738-4c3d-bc9f-9f5003408589\") " pod="openshift-dns/node-resolver-cw2sv" Oct 29 22:41:48.700648 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699717 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd35650d-2c53-4230-8518-c4cb05b59558-original-pull-secret\") pod \"global-pull-secret-syncer-ldgnp\" (UID: \"fd35650d-2c53-4230-8518-c4cb05b59558\") " pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:41:48.700648 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699740 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51fcfffe-f89d-4064-a0e3-45461cc62570-cni-binary-copy\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.700648 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699768 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-var-lib-kubelet\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.700648 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699795 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51fcfffe-f89d-4064-a0e3-45461cc62570-multus-daemon-config\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.700648 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699812 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c147a7be-d6f6-458e-8b0d-70da7a3c7026-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:41:48.700648 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699821 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-systemd-units\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.700648 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699829 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4916614f-8a34-491d-b4ad-bae72c14b140-metrics-client-ca\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.700648 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699847 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0061f2cf-c501-47b0-a89b-3e244f9c313d-konnectivity-ca\") pod \"konnectivity-agent-jzjsw\" (UID: \"0061f2cf-c501-47b0-a89b-3e244f9c313d\") " pod="kube-system/konnectivity-agent-jzjsw" Oct 29 22:41:48.700648 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699890 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-run-k8s-cni-cncf-io\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.700648 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699942 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-run-ovn-kubernetes\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.700648 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.699992 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c147a7be-d6f6-458e-8b0d-70da7a3c7026-system-cni-dir\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:41:48.700648 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:48.700036 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:41:48.701353 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.700055 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33360cd4-4738-4c3d-bc9f-9f5003408589-hosts-file\") pod \"node-resolver-cw2sv\" (UID: \"33360cd4-4738-4c3d-bc9f-9f5003408589\") " pod="openshift-dns/node-resolver-cw2sv" Oct 29 22:41:48.701353 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.700110 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-var-lib-cni-multus\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.701353 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.700279 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-run-netns\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.701353 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.700400 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-sysctl-conf\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.701353 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.700429 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33360cd4-4738-4c3d-bc9f-9f5003408589-tmp-dir\") pod \"node-resolver-cw2sv\" (UID: \"33360cd4-4738-4c3d-bc9f-9f5003408589\") " pod="openshift-dns/node-resolver-cw2sv" Oct 29 22:41:48.701353 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.700364 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c147a7be-d6f6-458e-8b0d-70da7a3c7026-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:41:48.701353 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.700459 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-device-dir\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:41:48.701353 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.700472 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-wtmp\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.701353 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.700500 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-etc-openvswitch\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.701353 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:48.700507 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:41:48.701353 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.700581 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-systemd-units\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.701353 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.700805 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99a7b227-aad8-414f-8668-70a5660d6a6e-host-slash\") pod \"iptables-alerter-bzvh4\" (UID: \"99a7b227-aad8-414f-8668-70a5660d6a6e\") " pod="openshift-network-operator/iptables-alerter-bzvh4" Oct 29 22:41:48.701353 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.700832 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-multus-conf-dir\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.701353 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.700845 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-var-lib-kubelet\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.701353 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.700872 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd8z9\" (UniqueName: \"kubernetes.io/projected/51fcfffe-f89d-4064-a0e3-45461cc62570-kube-api-access-wd8z9\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.701353 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.700904 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:41:48.701353 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.700928 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-var-lib-openvswitch\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.701353 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.700939 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-var-lib-kubelet\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.702139 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.700952 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-log-socket\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.702139 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.701496 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-var-lib-kubelet\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.702139 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.701608 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99a7b227-aad8-414f-8668-70a5660d6a6e-host-slash\") pod \"iptables-alerter-bzvh4\" (UID: \"99a7b227-aad8-414f-8668-70a5660d6a6e\") " pod="openshift-network-operator/iptables-alerter-bzvh4" Oct 29 22:41:48.702139 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.701664 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.702139 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.701681 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0061f2cf-c501-47b0-a89b-3e244f9c313d-konnectivity-ca\") pod \"konnectivity-agent-jzjsw\" (UID: \"0061f2cf-c501-47b0-a89b-3e244f9c313d\") " pod="kube-system/konnectivity-agent-jzjsw" Oct 29 22:41:48.702139 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.701766 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.702139 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.702065 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51fcfffe-f89d-4064-a0e3-45461cc62570-multus-daemon-config\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.702522 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.702223 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ea8559a-aa64-4d00-84c0-2c9405e32606-ovnkube-script-lib\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.702522 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.702237 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-var-lib-openvswitch\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.702522 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.702310 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:41:48.702522 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.702372 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-log-socket\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.702718 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.702603 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51fcfffe-f89d-4064-a0e3-45461cc62570-cni-binary-copy\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.702718 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:48.702692 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8ad4438-da5a-4f3a-bb5b-07c330aef792-metrics-certs podName:a8ad4438-da5a-4f3a-bb5b-07c330aef792 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:49.202666465 +0000 UTC m=+3.135044800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8ad4438-da5a-4f3a-bb5b-07c330aef792-metrics-certs") pod "network-metrics-daemon-29dzf" (UID: "a8ad4438-da5a-4f3a-bb5b-07c330aef792") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:41:48.702821 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:48.702731 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd35650d-2c53-4230-8518-c4cb05b59558-original-pull-secret podName:fd35650d-2c53-4230-8518-c4cb05b59558 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:49.202719774 +0000 UTC m=+3.135098095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd35650d-2c53-4230-8518-c4cb05b59558-original-pull-secret") pod "global-pull-secret-syncer-ldgnp" (UID: "fd35650d-2c53-4230-8518-c4cb05b59558") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:41:48.702821 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.702796 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ea8559a-aa64-4d00-84c0-2c9405e32606-ovnkube-script-lib\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.702821 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.702802 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-tmp\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.702951 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.702850 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmn55\" (UniqueName: \"kubernetes.io/projected/33360cd4-4738-4c3d-bc9f-9f5003408589-kube-api-access-fmn55\") pod \"node-resolver-cw2sv\" (UID: \"33360cd4-4738-4c3d-bc9f-9f5003408589\") " pod="openshift-dns/node-resolver-cw2sv" Oct 29 22:41:48.702951 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.702881 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-host\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.702951 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.702912 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-run-ovn\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.703095 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.702975 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-run-ovn\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.703095 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.702975 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-host\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.703095 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.703009 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c147a7be-d6f6-458e-8b0d-70da7a3c7026-cni-binary-copy\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:41:48.703095 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.703046 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-modprobe-d\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.703095 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.703073 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-tuned\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.703360 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.703104 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-tls\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.703360 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.703137 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.703360 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.703176 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-accelerators-collector-config\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.703360 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.703238 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d5dd\" (UniqueName: \"kubernetes.io/projected/4916614f-8a34-491d-b4ad-bae72c14b140-kube-api-access-6d5dd\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.703360 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.703264 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-lib-modules\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.703360 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.703294 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-cnibin\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.703360 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.703324 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-run-multus-certs\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.703688 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.703363 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-etc-kubernetes\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.703688 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.703400 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ea8559a-aa64-4d00-84c0-2c9405e32606-ovnkube-config\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.703688 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.703426 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/99a7b227-aad8-414f-8668-70a5660d6a6e-iptables-alerter-script\") pod \"iptables-alerter-bzvh4\" (UID: \"99a7b227-aad8-414f-8668-70a5660d6a6e\") " pod="openshift-network-operator/iptables-alerter-bzvh4" Oct 29 22:41:48.703688 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.703459 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb8pj\" (UniqueName: \"kubernetes.io/projected/c147a7be-d6f6-458e-8b0d-70da7a3c7026-kube-api-access-qb8pj\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:41:48.703688 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.703489 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-sys\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.703688 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.703518 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-multus-cni-dir\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.703688 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.703544 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-var-lib-cni-bin\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.703688 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.703574 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-hostroot\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.703688 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.703606 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6ll2\" (UniqueName: \"kubernetes.io/projected/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff-kube-api-access-f6ll2\") pod \"network-check-target-mbv7x\" (UID: \"b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff\") " pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:41:48.703688 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.703605 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c147a7be-d6f6-458e-8b0d-70da7a3c7026-cni-binary-copy\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:41:48.703688 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.703677 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-etc-kubernetes\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.704186 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.703928 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-multus-cni-dir\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.704612 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.704589 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-var-lib-cni-bin\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.707488 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.704659 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-hostroot\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.707488 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.704879 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0061f2cf-c501-47b0-a89b-3e244f9c313d-agent-certs\") pod \"konnectivity-agent-jzjsw\" (UID: \"0061f2cf-c501-47b0-a89b-3e244f9c313d\") " pod="kube-system/konnectivity-agent-jzjsw" Oct 29 22:41:48.707488 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.704987 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-modprobe-d\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.707488 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705018 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-sys\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.707488 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705031 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4845\" (UniqueName: \"kubernetes.io/projected/a8ad4438-da5a-4f3a-bb5b-07c330aef792-kube-api-access-x4845\") pod \"network-metrics-daemon-29dzf\" (UID: \"a8ad4438-da5a-4f3a-bb5b-07c330aef792\") " pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:41:48.707488 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705060 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-sysconfig\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.707488 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705067 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-lib-modules\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.707488 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705128 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-kubernetes\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.707488 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705157 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-systemd\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.707488 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705165 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-cnibin\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.707488 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705186 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-cni-bin\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.707488 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705239 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-registration-dir\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:41:48.707488 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705253 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-sysconfig\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.707488 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705277 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-slash\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.707488 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705309 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-run-openvswitch\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.707488 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705343 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45m2j\" (UniqueName: \"kubernetes.io/projected/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-kube-api-access-45m2j\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.707488 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705359 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ea8559a-aa64-4d00-84c0-2c9405e32606-ovnkube-config\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.707488 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705369 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4916614f-8a34-491d-b4ad-bae72c14b140-root\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.708906 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705401 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-sys-fs\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:41:48.708906 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705436 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klzkm\" (UniqueName: \"kubernetes.io/projected/6ea8559a-aa64-4d00-84c0-2c9405e32606-kube-api-access-klzkm\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.708906 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705444 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-run-multus-certs\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.708906 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705472 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-textfile\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.708906 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705533 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-tuned\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.708906 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705544 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4916614f-8a34-491d-b4ad-bae72c14b140-root\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.708906 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705540 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f99954e5-6dfe-457f-9a8f-22752798bb5d-serviceca\") pod \"node-ca-7kgv6\" (UID: \"f99954e5-6dfe-457f-9a8f-22752798bb5d\") " pod="openshift-image-registry/node-ca-7kgv6" Oct 29 22:41:48.708906 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705597 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-sysctl-d\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.708906 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705603 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-sys-fs\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:41:48.708906 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705626 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-registration-dir\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:41:48.708906 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705694 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-sysctl-d\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.708906 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705730 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-run\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.708906 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705758 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-slash\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.708906 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705761 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-system-cni-dir\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.708906 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705806 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-system-cni-dir\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.708906 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705817 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-multus-socket-dir-parent\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.708906 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.705855 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-run-netns\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.708906 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.706000 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-run-netns\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.709502 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.706055 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-run-openvswitch\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.709502 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.706261 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-accelerators-collector-config\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.709502 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.706316 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-textfile\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.709502 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.706341 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-kubernetes\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.709502 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.706381 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-cni-bin\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.709502 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.706434 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-multus-socket-dir-parent\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.709502 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.706485 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-systemd\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.709502 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.706870 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f99954e5-6dfe-457f-9a8f-22752798bb5d-serviceca\") pod \"node-ca-7kgv6\" (UID: \"f99954e5-6dfe-457f-9a8f-22752798bb5d\") " pod="openshift-image-registry/node-ca-7kgv6" Oct 29 22:41:48.709502 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.706899 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/99a7b227-aad8-414f-8668-70a5660d6a6e-iptables-alerter-script\") pod \"iptables-alerter-bzvh4\" (UID: \"99a7b227-aad8-414f-8668-70a5660d6a6e\") " pod="openshift-network-operator/iptables-alerter-bzvh4" Oct 29 22:41:48.709502 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.706936 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-run\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.709502 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.708343 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.710676 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.710656 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ea8559a-aa64-4d00-84c0-2c9405e32606-ovn-node-metrics-cert\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.711231 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.711194 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-tls\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.723675 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.723647 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvp94\" (UniqueName: \"kubernetes.io/projected/99a7b227-aad8-414f-8668-70a5660d6a6e-kube-api-access-qvp94\") pod \"iptables-alerter-bzvh4\" (UID: \"99a7b227-aad8-414f-8668-70a5660d6a6e\") " pod="openshift-network-operator/iptables-alerter-bzvh4" Oct 29 22:41:48.723791 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.723655 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsddk\" (UniqueName: \"kubernetes.io/projected/f99954e5-6dfe-457f-9a8f-22752798bb5d-kube-api-access-hsddk\") pod \"node-ca-7kgv6\" (UID: \"f99954e5-6dfe-457f-9a8f-22752798bb5d\") " pod="openshift-image-registry/node-ca-7kgv6" Oct 29 22:41:48.723963 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.723909 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w9d9\" (UniqueName: \"kubernetes.io/projected/af139c4a-79a2-47b4-95e4-493e3430be98-kube-api-access-9w9d9\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:41:48.725452 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.725427 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd8z9\" (UniqueName: \"kubernetes.io/projected/51fcfffe-f89d-4064-a0e3-45461cc62570-kube-api-access-wd8z9\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.725925 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.725904 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmn55\" (UniqueName: \"kubernetes.io/projected/33360cd4-4738-4c3d-bc9f-9f5003408589-kube-api-access-fmn55\") pod \"node-resolver-cw2sv\" (UID: \"33360cd4-4738-4c3d-bc9f-9f5003408589\") " pod="openshift-dns/node-resolver-cw2sv" Oct 29 22:41:48.730628 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:48.730606 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:41:48.730628 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:48.730627 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:41:48.730777 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:48.730642 2577 projected.go:196] Error preparing data for projected volume kube-api-access-f6ll2 for pod openshift-network-diagnostics/network-check-target-mbv7x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:41:48.730777 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:48.730693 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff-kube-api-access-f6ll2 podName:b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff nodeName:}" failed. No retries permitted until 2026-10-29 22:41:49.230679955 +0000 UTC m=+3.163058290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f6ll2" (UniqueName: "kubernetes.io/projected/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff-kube-api-access-f6ll2") pod "network-check-target-mbv7x" (UID: "b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:41:48.732692 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.732672 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d5dd\" (UniqueName: \"kubernetes.io/projected/4916614f-8a34-491d-b4ad-bae72c14b140-kube-api-access-6d5dd\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.732785 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.732689 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4845\" (UniqueName: \"kubernetes.io/projected/a8ad4438-da5a-4f3a-bb5b-07c330aef792-kube-api-access-x4845\") pod \"network-metrics-daemon-29dzf\" (UID: \"a8ad4438-da5a-4f3a-bb5b-07c330aef792\") " pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:41:48.732942 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.732906 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb8pj\" (UniqueName: \"kubernetes.io/projected/c147a7be-d6f6-458e-8b0d-70da7a3c7026-kube-api-access-qb8pj\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:41:48.734476 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.734460 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45m2j\" (UniqueName: \"kubernetes.io/projected/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-kube-api-access-45m2j\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.737599 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.737581 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klzkm\" (UniqueName: \"kubernetes.io/projected/6ea8559a-aa64-4d00-84c0-2c9405e32606-kube-api-access-klzkm\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.814565 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.814537 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vdz2c" Oct 29 22:41:48.826615 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:48.826581 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fcfffe_f89d_4064_a0e3_45461cc62570.slice/crio-5738903c517750fdd3189dd5853f08e305912b8e74f0504ec99dde0ffbb5d401 WatchSource:0}: Error finding container 5738903c517750fdd3189dd5853f08e305912b8e74f0504ec99dde0ffbb5d401: Status 404 returned error can't find the container with id 5738903c517750fdd3189dd5853f08e305912b8e74f0504ec99dde0ffbb5d401 Oct 29 22:41:48.844028 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.844005 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:41:48.849000 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.848977 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7kgv6" Oct 29 22:41:48.849123 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.849097 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:41:48.852845 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.852823 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cw2sv" Oct 29 22:41:48.854322 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:48.854294 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc147a7be_d6f6_458e_8b0d_70da7a3c7026.slice/crio-256d11bdaba8884eeb5c071b63c56ee8fbe9b4f8b79ca262d8b6727b9d4e3353 WatchSource:0}: Error finding container 256d11bdaba8884eeb5c071b63c56ee8fbe9b4f8b79ca262d8b6727b9d4e3353: Status 404 returned error can't find the container with id 256d11bdaba8884eeb5c071b63c56ee8fbe9b4f8b79ca262d8b6727b9d4e3353 Oct 29 22:41:48.861034 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.860895 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:41:48.864296 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:48.864274 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf139c4a_79a2_47b4_95e4_493e3430be98.slice/crio-be01642a04bd5f86f0f4682b268dea238129b34cadf8482f8ab0c03cdf7f27b5 WatchSource:0}: Error finding container be01642a04bd5f86f0f4682b268dea238129b34cadf8482f8ab0c03cdf7f27b5: Status 404 returned error can't find the container with id be01642a04bd5f86f0f4682b268dea238129b34cadf8482f8ab0c03cdf7f27b5 Oct 29 22:41:48.865140 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:48.865110 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf99954e5_6dfe_457f_9a8f_22752798bb5d.slice/crio-9df89bff6b79e8cfaddaf04603087a596c31dbbb8591f9c9d0abb9e16da76f5d WatchSource:0}: Error finding container 9df89bff6b79e8cfaddaf04603087a596c31dbbb8591f9c9d0abb9e16da76f5d: Status 404 returned error can't find the container with id 9df89bff6b79e8cfaddaf04603087a596c31dbbb8591f9c9d0abb9e16da76f5d Oct 29 22:41:48.869505 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:48.869472 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33360cd4_4738_4c3d_bc9f_9f5003408589.slice/crio-5fe6c4845b6d870cedbb00207a211c8cb9216002c9047e6175f2e691cac77de4 WatchSource:0}: Error finding container 5fe6c4845b6d870cedbb00207a211c8cb9216002c9047e6175f2e691cac77de4: Status 404 returned error can't find the container with id 5fe6c4845b6d870cedbb00207a211c8cb9216002c9047e6175f2e691cac77de4 Oct 29 22:41:48.874430 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:48.874407 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode98b6bfb_2f6b_4d7a_9541_7dfbb8a88fd1.slice/crio-7ae03d68650335aef8a8c76122f149c51f4d0a635fefabced4459527dce2ceae WatchSource:0}: Error finding container 7ae03d68650335aef8a8c76122f149c51f4d0a635fefabced4459527dce2ceae: Status 404 returned error can't find the container with id 7ae03d68650335aef8a8c76122f149c51f4d0a635fefabced4459527dce2ceae Oct 29 22:41:48.880642 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.880618 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jzjsw" Oct 29 22:41:48.888298 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.888274 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:41:48.890363 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.890343 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bzvh4" Oct 29 22:41:48.896027 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:48.895879 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:41:48.907265 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:48.907238 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea8559a_aa64_4d00_84c0_2c9405e32606.slice/crio-8258a9c17fdfce96ec8fdaa0b7d8b53e69dd050c3f578c21279eca26a2ded146 WatchSource:0}: Error finding container 8258a9c17fdfce96ec8fdaa0b7d8b53e69dd050c3f578c21279eca26a2ded146: Status 404 returned error can't find the container with id 8258a9c17fdfce96ec8fdaa0b7d8b53e69dd050c3f578c21279eca26a2ded146 Oct 29 22:41:48.911250 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:48.911225 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a7b227_aad8_414f_8668_70a5660d6a6e.slice/crio-aa2507ccc88a01b0ef9b0e5bb4d7b8c4ab759816bfdac374fac1e12a2437e2a2 WatchSource:0}: Error finding container aa2507ccc88a01b0ef9b0e5bb4d7b8c4ab759816bfdac374fac1e12a2437e2a2: Status 404 returned error can't find the container with id aa2507ccc88a01b0ef9b0e5bb4d7b8c4ab759816bfdac374fac1e12a2437e2a2 Oct 29 22:41:48.913329 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:41:48.913306 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4916614f_8a34_491d_b4ad_bae72c14b140.slice/crio-a32d691bc1c503e421d0c7cade812b5ed084aa8b5ffcda1cfa550ec243d2c1e4 WatchSource:0}: Error finding container a32d691bc1c503e421d0c7cade812b5ed084aa8b5ffcda1cfa550ec243d2c1e4: Status 404 returned error can't find the container with id a32d691bc1c503e421d0c7cade812b5ed084aa8b5ffcda1cfa550ec243d2c1e4 Oct 29 22:41:49.208903 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:49.208676 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8ad4438-da5a-4f3a-bb5b-07c330aef792-metrics-certs\") pod \"network-metrics-daemon-29dzf\" (UID: \"a8ad4438-da5a-4f3a-bb5b-07c330aef792\") " pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:41:49.209080 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:49.208934 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:41:49.209080 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:49.209010 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8ad4438-da5a-4f3a-bb5b-07c330aef792-metrics-certs podName:a8ad4438-da5a-4f3a-bb5b-07c330aef792 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:50.20899057 +0000 UTC m=+4.141368910 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8ad4438-da5a-4f3a-bb5b-07c330aef792-metrics-certs") pod "network-metrics-daemon-29dzf" (UID: "a8ad4438-da5a-4f3a-bb5b-07c330aef792") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:41:49.209080 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:49.209019 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:41:49.209080 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:49.208939 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd35650d-2c53-4230-8518-c4cb05b59558-original-pull-secret\") pod \"global-pull-secret-syncer-ldgnp\" (UID: \"fd35650d-2c53-4230-8518-c4cb05b59558\") " pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:41:49.209080 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:49.209068 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd35650d-2c53-4230-8518-c4cb05b59558-original-pull-secret podName:fd35650d-2c53-4230-8518-c4cb05b59558 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:50.209053669 +0000 UTC m=+4.141432013 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd35650d-2c53-4230-8518-c4cb05b59558-original-pull-secret") pod "global-pull-secret-syncer-ldgnp" (UID: "fd35650d-2c53-4230-8518-c4cb05b59558") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:41:49.309997 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:49.309883 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6ll2\" (UniqueName: \"kubernetes.io/projected/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff-kube-api-access-f6ll2\") pod \"network-check-target-mbv7x\" (UID: \"b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff\") " pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:41:49.310160 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:49.310110 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:41:49.310160 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:49.310129 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:41:49.310160 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:49.310141 2577 projected.go:196] Error preparing data for projected volume kube-api-access-f6ll2 for pod openshift-network-diagnostics/network-check-target-mbv7x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:41:49.310333 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:49.310216 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff-kube-api-access-f6ll2 podName:b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff nodeName:}" failed. No retries permitted until 2026-10-29 22:41:50.31018036 +0000 UTC m=+4.242558696 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f6ll2" (UniqueName: "kubernetes.io/projected/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff-kube-api-access-f6ll2") pod "network-check-target-mbv7x" (UID: "b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:41:49.545301 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:49.545158 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:36:48 +0000 UTC" deadline="2028-04-21 00:24:09.663483154 +0000 UTC" Oct 29 22:41:49.545301 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:49.545242 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12937h42m20.118244864s" Oct 29 22:41:49.578362 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:49.578289 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vdz2c" event={"ID":"51fcfffe-f89d-4064-a0e3-45461cc62570","Type":"ContainerStarted","Data":"5738903c517750fdd3189dd5853f08e305912b8e74f0504ec99dde0ffbb5d401"} Oct 29 22:41:49.594524 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:49.594483 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-252.ec2.internal" event={"ID":"83fee9c1d2770428373c9940b59a21be","Type":"ContainerStarted","Data":"ceda17f91eddb9cc7addb9a8d9bb8710f7cdc2fb336929d04e0fca0571beaf63"} Oct 29 22:41:49.605605 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:49.605569 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-shnd6" event={"ID":"4916614f-8a34-491d-b4ad-bae72c14b140","Type":"ContainerStarted","Data":"a32d691bc1c503e421d0c7cade812b5ed084aa8b5ffcda1cfa550ec243d2c1e4"} Oct 29 22:41:49.621967 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:49.621932 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bzvh4" event={"ID":"99a7b227-aad8-414f-8668-70a5660d6a6e","Type":"ContainerStarted","Data":"aa2507ccc88a01b0ef9b0e5bb4d7b8c4ab759816bfdac374fac1e12a2437e2a2"} Oct 29 22:41:49.629835 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:49.629806 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" event={"ID":"6ea8559a-aa64-4d00-84c0-2c9405e32606","Type":"ContainerStarted","Data":"8258a9c17fdfce96ec8fdaa0b7d8b53e69dd050c3f578c21279eca26a2ded146"} Oct 29 22:41:49.633785 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:49.633714 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-252.ec2.internal" podStartSLOduration=1.6336975310000001 podStartE2EDuration="1.633697531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:49.633018686 +0000 UTC m=+3.565397033" watchObservedRunningTime="2026-10-29 22:41:49.633697531 +0000 UTC m=+3.566075875" Oct 29 22:41:49.647711 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:49.647630 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jzjsw" event={"ID":"0061f2cf-c501-47b0-a89b-3e244f9c313d","Type":"ContainerStarted","Data":"ef45efa92af3dab51ceeb174fdbcec84aa5201e24deda9837a403933bd21f0a0"} Oct 29 22:41:49.662580 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:49.662493 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cw2sv" event={"ID":"33360cd4-4738-4c3d-bc9f-9f5003408589","Type":"ContainerStarted","Data":"5fe6c4845b6d870cedbb00207a211c8cb9216002c9047e6175f2e691cac77de4"} Oct 29 22:41:49.674583 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:49.674520 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" event={"ID":"af139c4a-79a2-47b4-95e4-493e3430be98","Type":"ContainerStarted","Data":"be01642a04bd5f86f0f4682b268dea238129b34cadf8482f8ab0c03cdf7f27b5"} Oct 29 22:41:49.680286 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:49.680237 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdtdb" event={"ID":"c147a7be-d6f6-458e-8b0d-70da7a3c7026","Type":"ContainerStarted","Data":"256d11bdaba8884eeb5c071b63c56ee8fbe9b4f8b79ca262d8b6727b9d4e3353"} Oct 29 22:41:49.683840 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:49.683217 2577 generic.go:356] "Generic (PLEG): container finished" podID="3f8a08c2f1d6fb4fc298c3bba521bafd" containerID="bbfd3a9d9702654a11c3b592516fa8f84fdbca5a49d8fc28286ce806bbb0f281" exitCode=0 Oct 29 22:41:49.683840 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:49.683304 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" event={"ID":"3f8a08c2f1d6fb4fc298c3bba521bafd","Type":"ContainerDied","Data":"bbfd3a9d9702654a11c3b592516fa8f84fdbca5a49d8fc28286ce806bbb0f281"} Oct 29 22:41:49.698650 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:49.698597 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" event={"ID":"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1","Type":"ContainerStarted","Data":"7ae03d68650335aef8a8c76122f149c51f4d0a635fefabced4459527dce2ceae"} Oct 29 22:41:49.711687 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:49.711597 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7kgv6" event={"ID":"f99954e5-6dfe-457f-9a8f-22752798bb5d","Type":"ContainerStarted","Data":"9df89bff6b79e8cfaddaf04603087a596c31dbbb8591f9c9d0abb9e16da76f5d"} Oct 29 22:41:50.217613 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:50.216733 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd35650d-2c53-4230-8518-c4cb05b59558-original-pull-secret\") pod \"global-pull-secret-syncer-ldgnp\" (UID: \"fd35650d-2c53-4230-8518-c4cb05b59558\") " pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:41:50.217613 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:50.216828 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8ad4438-da5a-4f3a-bb5b-07c330aef792-metrics-certs\") pod \"network-metrics-daemon-29dzf\" (UID: \"a8ad4438-da5a-4f3a-bb5b-07c330aef792\") " pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:41:50.217613 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:50.216943 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:41:50.217613 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:50.216998 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8ad4438-da5a-4f3a-bb5b-07c330aef792-metrics-certs podName:a8ad4438-da5a-4f3a-bb5b-07c330aef792 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:52.216981741 +0000 UTC m=+6.149360066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8ad4438-da5a-4f3a-bb5b-07c330aef792-metrics-certs") pod "network-metrics-daemon-29dzf" (UID: "a8ad4438-da5a-4f3a-bb5b-07c330aef792") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:41:50.217613 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:50.217454 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:41:50.217613 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:50.217506 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd35650d-2c53-4230-8518-c4cb05b59558-original-pull-secret podName:fd35650d-2c53-4230-8518-c4cb05b59558 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:52.217492061 +0000 UTC m=+6.149870396 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd35650d-2c53-4230-8518-c4cb05b59558-original-pull-secret") pod "global-pull-secret-syncer-ldgnp" (UID: "fd35650d-2c53-4230-8518-c4cb05b59558") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:41:50.318050 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:50.318010 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6ll2\" (UniqueName: \"kubernetes.io/projected/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff-kube-api-access-f6ll2\") pod \"network-check-target-mbv7x\" (UID: \"b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff\") " pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:41:50.318264 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:50.318246 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:41:50.318351 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:50.318275 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:41:50.318351 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:50.318287 2577 projected.go:196] Error preparing data for projected volume kube-api-access-f6ll2 for pod openshift-network-diagnostics/network-check-target-mbv7x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:41:50.318351 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:50.318343 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff-kube-api-access-f6ll2 podName:b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff nodeName:}" failed. No retries permitted until 2026-10-29 22:41:52.318326074 +0000 UTC m=+6.250704402 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f6ll2" (UniqueName: "kubernetes.io/projected/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff-kube-api-access-f6ll2") pod "network-check-target-mbv7x" (UID: "b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:41:50.546350 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:50.546099 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:36:48 +0000 UTC" deadline="2028-05-04 04:34:53.574634282 +0000 UTC" Oct 29 22:41:50.546350 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:50.546139 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13253h53m3.028500253s" Oct 29 22:41:50.559947 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:50.559214 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:41:50.559947 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:50.559344 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbv7x" podUID="b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff" Oct 29 22:41:50.559947 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:50.559791 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:41:50.559947 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:50.559899 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dzf" podUID="a8ad4438-da5a-4f3a-bb5b-07c330aef792" Oct 29 22:41:50.560469 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:50.560333 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:41:50.560469 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:50.560432 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldgnp" podUID="fd35650d-2c53-4230-8518-c4cb05b59558" Oct 29 22:41:50.751445 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:50.751175 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" event={"ID":"3f8a08c2f1d6fb4fc298c3bba521bafd","Type":"ContainerStarted","Data":"8be65d7a8dfe56c1cdee7f5844d005504615b923195713109b0672866c8fd3b7"} Oct 29 22:41:51.767384 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:51.767307 2577 generic.go:356] "Generic (PLEG): container finished" podID="4916614f-8a34-491d-b4ad-bae72c14b140" containerID="a0b917804dc11cc0b5b3024c5019a3f172efdf89df4089f20c127820ffcaa56b" exitCode=0 Oct 29 22:41:51.768434 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:51.768402 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-shnd6" event={"ID":"4916614f-8a34-491d-b4ad-bae72c14b140","Type":"ContainerDied","Data":"a0b917804dc11cc0b5b3024c5019a3f172efdf89df4089f20c127820ffcaa56b"} Oct 29 22:41:51.817454 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:51.816624 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" podStartSLOduration=3.816591425 podStartE2EDuration="3.816591425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:50.791536666 +0000 UTC m=+4.723915011" watchObservedRunningTime="2026-10-29 22:41:51.816591425 +0000 UTC m=+5.748969770" Oct 29 22:41:52.235590 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:52.234777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8ad4438-da5a-4f3a-bb5b-07c330aef792-metrics-certs\") pod \"network-metrics-daemon-29dzf\" (UID: \"a8ad4438-da5a-4f3a-bb5b-07c330aef792\") " pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:41:52.235590 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:52.234843 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd35650d-2c53-4230-8518-c4cb05b59558-original-pull-secret\") pod \"global-pull-secret-syncer-ldgnp\" (UID: \"fd35650d-2c53-4230-8518-c4cb05b59558\") " pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:41:52.235590 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:52.234979 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:41:52.235590 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:52.235042 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd35650d-2c53-4230-8518-c4cb05b59558-original-pull-secret podName:fd35650d-2c53-4230-8518-c4cb05b59558 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:56.235023599 +0000 UTC m=+10.167401938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd35650d-2c53-4230-8518-c4cb05b59558-original-pull-secret") pod "global-pull-secret-syncer-ldgnp" (UID: "fd35650d-2c53-4230-8518-c4cb05b59558") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:41:52.235590 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:52.235459 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:41:52.235590 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:52.235507 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8ad4438-da5a-4f3a-bb5b-07c330aef792-metrics-certs podName:a8ad4438-da5a-4f3a-bb5b-07c330aef792 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:56.235493878 +0000 UTC m=+10.167872199 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8ad4438-da5a-4f3a-bb5b-07c330aef792-metrics-certs") pod "network-metrics-daemon-29dzf" (UID: "a8ad4438-da5a-4f3a-bb5b-07c330aef792") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:41:52.336194 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:52.335510 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6ll2\" (UniqueName: \"kubernetes.io/projected/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff-kube-api-access-f6ll2\") pod \"network-check-target-mbv7x\" (UID: \"b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff\") " pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:41:52.336194 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:52.335741 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:41:52.336194 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:52.335764 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:41:52.336194 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:52.335779 2577 projected.go:196] Error preparing data for projected volume kube-api-access-f6ll2 for pod openshift-network-diagnostics/network-check-target-mbv7x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:41:52.336194 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:52.335836 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff-kube-api-access-f6ll2 podName:b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff nodeName:}" failed. No retries permitted until 2026-10-29 22:41:56.335819686 +0000 UTC m=+10.268198013 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f6ll2" (UniqueName: "kubernetes.io/projected/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff-kube-api-access-f6ll2") pod "network-check-target-mbv7x" (UID: "b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:41:52.561947 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:52.561031 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:41:52.561947 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:52.561162 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldgnp" podUID="fd35650d-2c53-4230-8518-c4cb05b59558" Oct 29 22:41:52.561947 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:52.561555 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:41:52.561947 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:52.561649 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbv7x" podUID="b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff" Oct 29 22:41:52.561947 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:52.561719 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:41:52.561947 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:52.561806 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dzf" podUID="a8ad4438-da5a-4f3a-bb5b-07c330aef792" Oct 29 22:41:52.779655 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:52.778851 2577 generic.go:356] "Generic (PLEG): container finished" podID="c147a7be-d6f6-458e-8b0d-70da7a3c7026" containerID="60141d85a7f0048d0ba923db485e1427d138632ff50d9568cebfd52e82f79f20" exitCode=0 Oct 29 22:41:52.779655 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:52.778946 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdtdb" event={"ID":"c147a7be-d6f6-458e-8b0d-70da7a3c7026","Type":"ContainerDied","Data":"60141d85a7f0048d0ba923db485e1427d138632ff50d9568cebfd52e82f79f20"} Oct 29 22:41:52.783918 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:52.783890 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-shnd6" event={"ID":"4916614f-8a34-491d-b4ad-bae72c14b140","Type":"ContainerStarted","Data":"139ecb574c91c56e8a4bc5f5b97b076ce1c5ec414e3a3f6c528fc93378d0fac8"} Oct 29 22:41:52.783997 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:52.783942 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-shnd6" event={"ID":"4916614f-8a34-491d-b4ad-bae72c14b140","Type":"ContainerStarted","Data":"784870c243e949b7778d92fbbad2a30f0ceba6b46c3d0f61c2ef11093e356f80"} Oct 29 22:41:52.870680 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:52.870615 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-shnd6" podStartSLOduration=3.280875853 podStartE2EDuration="5.87059488s" totalImagesPullingTime="2.498252212s" totalInitContainerRuntime="91.466815ms" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:52.870464169 +0000 UTC m=+6.802842518" watchObservedRunningTime="2026-10-29 22:41:52.87059488 +0000 UTC m=+6.802973259" Oct 29 22:41:53.792174 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:53.792125 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jzjsw" event={"ID":"0061f2cf-c501-47b0-a89b-3e244f9c313d","Type":"ContainerStarted","Data":"b4c9491447cb802324b5bb1a66d6691a0f762a2010ec012378d1f04eab08446c"} Oct 29 22:41:54.560271 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:54.560240 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:41:54.560412 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:54.560308 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:41:54.560412 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:54.560328 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:41:54.560534 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:54.560441 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dzf" podUID="a8ad4438-da5a-4f3a-bb5b-07c330aef792" Oct 29 22:41:54.560740 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:54.560684 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbv7x" podUID="b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff" Oct 29 22:41:54.560816 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:54.560764 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldgnp" podUID="fd35650d-2c53-4230-8518-c4cb05b59558" Oct 29 22:41:54.799410 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:54.799364 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" event={"ID":"af139c4a-79a2-47b4-95e4-493e3430be98","Type":"ContainerStarted","Data":"e61a82e98c68cdd3ca217df5cefe3d36ff83beb8d1f1425bcb2e3b957751f6be"} Oct 29 22:41:54.804983 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:54.804953 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7kgv6" event={"ID":"f99954e5-6dfe-457f-9a8f-22752798bb5d","Type":"ContainerStarted","Data":"d77f7e7118afc39450a9099102e62ce9db482e87f6270f1f2a938dc3d64560f6"} Oct 29 22:41:54.844875 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:54.844809 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jzjsw" podStartSLOduration=5.007390991 podStartE2EDuration="8.844787494s" totalImagesPullingTime="3.837396503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:53.833506715 +0000 UTC m=+7.765885059" watchObservedRunningTime="2026-10-29 22:41:54.844787494 +0000 UTC m=+8.777165856" Oct 29 22:41:56.202670 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:56.202644 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 29 22:41:56.271623 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:56.270800 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8ad4438-da5a-4f3a-bb5b-07c330aef792-metrics-certs\") pod \"network-metrics-daemon-29dzf\" (UID: \"a8ad4438-da5a-4f3a-bb5b-07c330aef792\") " pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:41:56.271623 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:56.270871 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd35650d-2c53-4230-8518-c4cb05b59558-original-pull-secret\") pod \"global-pull-secret-syncer-ldgnp\" (UID: \"fd35650d-2c53-4230-8518-c4cb05b59558\") " pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:41:56.271623 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:56.271042 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:41:56.271623 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:56.271104 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd35650d-2c53-4230-8518-c4cb05b59558-original-pull-secret podName:fd35650d-2c53-4230-8518-c4cb05b59558 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:04.271085924 +0000 UTC m=+18.203464246 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd35650d-2c53-4230-8518-c4cb05b59558-original-pull-secret") pod "global-pull-secret-syncer-ldgnp" (UID: "fd35650d-2c53-4230-8518-c4cb05b59558") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:41:56.271623 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:56.271502 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:41:56.271623 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:56.271549 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8ad4438-da5a-4f3a-bb5b-07c330aef792-metrics-certs podName:a8ad4438-da5a-4f3a-bb5b-07c330aef792 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:04.271535414 +0000 UTC m=+18.203913735 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8ad4438-da5a-4f3a-bb5b-07c330aef792-metrics-certs") pod "network-metrics-daemon-29dzf" (UID: "a8ad4438-da5a-4f3a-bb5b-07c330aef792") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:41:56.372951 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:56.372603 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6ll2\" (UniqueName: \"kubernetes.io/projected/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff-kube-api-access-f6ll2\") pod \"network-check-target-mbv7x\" (UID: \"b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff\") " pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:41:56.372951 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:56.372801 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:41:56.372951 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:56.372819 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:41:56.372951 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:56.372830 2577 projected.go:196] Error preparing data for projected volume kube-api-access-f6ll2 for pod openshift-network-diagnostics/network-check-target-mbv7x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:41:56.372951 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:56.372885 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff-kube-api-access-f6ll2 podName:b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff nodeName:}" failed. No retries permitted until 2026-10-29 22:42:04.37286482 +0000 UTC m=+18.305243155 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f6ll2" (UniqueName: "kubernetes.io/projected/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff-kube-api-access-f6ll2") pod "network-check-target-mbv7x" (UID: "b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:41:56.558789 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:56.558652 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:41:56.202665584Z","UUID":"613b12ed-f3f3-48ee-b8c3-0875e5b92f0a","Handler":null,"Name":"","Endpoint":""} Oct 29 22:41:56.565083 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:56.565055 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 29 22:41:56.565237 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:56.565181 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 29 22:41:56.567457 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:56.565499 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:41:56.567457 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:56.565622 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:41:56.567457 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:56.565672 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:41:56.567457 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:56.565816 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dzf" podUID="a8ad4438-da5a-4f3a-bb5b-07c330aef792" Oct 29 22:41:56.567457 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:56.566237 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbv7x" podUID="b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff" Oct 29 22:41:56.567457 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:56.566365 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldgnp" podUID="fd35650d-2c53-4230-8518-c4cb05b59558" Oct 29 22:41:56.812145 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:56.811461 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vdz2c" event={"ID":"51fcfffe-f89d-4064-a0e3-45461cc62570","Type":"ContainerStarted","Data":"3a4d4f9ddb202c613b9d8d7787181482f65f2a57937c3287b458fabca432a73f"} Oct 29 22:41:56.814713 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:56.814665 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" event={"ID":"af139c4a-79a2-47b4-95e4-493e3430be98","Type":"ContainerStarted","Data":"62604139c9cb96387c1ea508514e2013cc2396441aa56957e22b706a6c2d4694"} Oct 29 22:41:56.855288 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:56.855222 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7kgv6" podStartSLOduration=5.861983449 podStartE2EDuration="10.855182119s" totalImagesPullingTime="4.99319867s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:54.845376858 +0000 UTC m=+8.777755234" watchObservedRunningTime="2026-10-29 22:41:56.855182119 +0000 UTC m=+10.787560468" Oct 29 22:41:56.855730 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:56.855683 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-vdz2c" podStartSLOduration=3.407629133 podStartE2EDuration="10.855670413s" totalImagesPullingTime="7.44804128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:56.85477005 +0000 UTC m=+10.787148419" watchObservedRunningTime="2026-10-29 22:41:56.855670413 +0000 UTC m=+10.788048757" Oct 29 22:41:57.122787 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:57.122631 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jzjsw" Oct 29 22:41:57.124540 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:57.124519 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jzjsw" Oct 29 22:41:57.820000 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:57.819920 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cw2sv" event={"ID":"33360cd4-4738-4c3d-bc9f-9f5003408589","Type":"ContainerStarted","Data":"d09122eb81c5eb2abd8d8c4175f094ae434cfad4bb4ca7fadc6f66c400867949"} Oct 29 22:41:57.822608 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:57.822580 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" event={"ID":"af139c4a-79a2-47b4-95e4-493e3430be98","Type":"ContainerStarted","Data":"609b71d9c70abf9876b0051130a26d9f0f94053be9a8f7d3dde3f63ab1cd9844"} Oct 29 22:41:57.825352 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:57.825325 2577 generic.go:356] "Generic (PLEG): container finished" podID="c147a7be-d6f6-458e-8b0d-70da7a3c7026" containerID="1268dd417c60587166b2e56108b0e7223294c5abf3be1f146c24957b332035ae" exitCode=0 Oct 29 22:41:57.825464 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:57.825436 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdtdb" event={"ID":"c147a7be-d6f6-458e-8b0d-70da7a3c7026","Type":"ContainerDied","Data":"1268dd417c60587166b2e56108b0e7223294c5abf3be1f146c24957b332035ae"} Oct 29 22:41:57.859688 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:57.859625 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-cw2sv" podStartSLOduration=3.867106724 podStartE2EDuration="11.859606121s" totalImagesPullingTime="7.992499397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:57.859295377 +0000 UTC m=+11.791673720" watchObservedRunningTime="2026-10-29 22:41:57.859606121 +0000 UTC m=+11.791984466" Oct 29 22:41:57.950579 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:57.950525 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" podStartSLOduration=3.983551758 podStartE2EDuration="11.950512297s" totalImagesPullingTime="7.966960539s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:57.905377354 +0000 UTC m=+11.837755696" watchObservedRunningTime="2026-10-29 22:41:57.950512297 +0000 UTC m=+11.882890640" Oct 29 22:41:58.559504 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:58.559466 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:41:58.559688 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:58.559617 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbv7x" podUID="b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff" Oct 29 22:41:58.560054 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:58.560030 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:41:58.560180 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:58.560145 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dzf" podUID="a8ad4438-da5a-4f3a-bb5b-07c330aef792" Oct 29 22:41:58.560256 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:58.560241 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:41:58.560331 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:41:58.560313 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldgnp" podUID="fd35650d-2c53-4230-8518-c4cb05b59558" Oct 29 22:41:58.829664 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:58.829592 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bzvh4" event={"ID":"99a7b227-aad8-414f-8668-70a5660d6a6e","Type":"ContainerStarted","Data":"68c3b8843552e2e5b5774ab2ffae1e158726e9f09ba06439a43181bef6794288"} Oct 29 22:41:58.869378 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:58.869323 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bzvh4" podStartSLOduration=4.6798589889999995 podStartE2EDuration="12.869309207s" totalImagesPullingTime="8.189450218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:58.868762277 +0000 UTC m=+12.801140621" watchObservedRunningTime="2026-10-29 22:41:58.869309207 +0000 UTC m=+12.801687549" Oct 29 22:41:59.835248 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:59.835007 2577 generic.go:356] "Generic (PLEG): container finished" podID="c147a7be-d6f6-458e-8b0d-70da7a3c7026" containerID="c34262dbde983262b88881152c6f97adc24862fc36c4c23c49997febb8237be7" exitCode=0 Oct 29 22:41:59.835831 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:59.835310 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdtdb" event={"ID":"c147a7be-d6f6-458e-8b0d-70da7a3c7026","Type":"ContainerDied","Data":"c34262dbde983262b88881152c6f97adc24862fc36c4c23c49997febb8237be7"} Oct 29 22:41:59.837904 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:59.837877 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" event={"ID":"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1","Type":"ContainerStarted","Data":"f805fd11ede03d166d26a60fc66f98e4c5331dfe7853bdc0142c454776d25365"} Oct 29 22:41:59.950630 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:41:59.950583 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" podStartSLOduration=3.548068034 podStartE2EDuration="13.950568772s" totalImagesPullingTime="10.402500738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:59.950080154 +0000 UTC m=+13.882458498" watchObservedRunningTime="2026-10-29 22:41:59.950568772 +0000 UTC m=+13.882947114" Oct 29 22:42:00.558911 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:00.558871 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:42:00.558911 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:00.558902 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:42:00.558911 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:00.558918 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:42:00.559175 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:00.559007 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dzf" podUID="a8ad4438-da5a-4f3a-bb5b-07c330aef792" Oct 29 22:42:00.559175 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:00.559079 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbv7x" podUID="b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff" Oct 29 22:42:00.559175 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:00.559155 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldgnp" podUID="fd35650d-2c53-4230-8518-c4cb05b59558" Oct 29 22:42:00.843581 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:00.843539 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdtdb" event={"ID":"c147a7be-d6f6-458e-8b0d-70da7a3c7026","Type":"ContainerStarted","Data":"6467ad9bfa7075c9a0ebfc00d2a1dbf90d197c6210eedb39840dd8881ac9963f"} Oct 29 22:42:01.848185 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:01.848140 2577 generic.go:356] "Generic (PLEG): container finished" podID="c147a7be-d6f6-458e-8b0d-70da7a3c7026" containerID="6467ad9bfa7075c9a0ebfc00d2a1dbf90d197c6210eedb39840dd8881ac9963f" exitCode=0 Oct 29 22:42:01.848614 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:01.848217 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdtdb" event={"ID":"c147a7be-d6f6-458e-8b0d-70da7a3c7026","Type":"ContainerDied","Data":"6467ad9bfa7075c9a0ebfc00d2a1dbf90d197c6210eedb39840dd8881ac9963f"} Oct 29 22:42:02.558980 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:02.558563 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:42:02.558980 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:02.558605 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:42:02.558980 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:02.558573 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:42:02.558980 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:02.558684 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbv7x" podUID="b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff" Oct 29 22:42:02.558980 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:02.558786 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dzf" podUID="a8ad4438-da5a-4f3a-bb5b-07c330aef792" Oct 29 22:42:02.558980 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:02.558851 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldgnp" podUID="fd35650d-2c53-4230-8518-c4cb05b59558" Oct 29 22:42:03.855247 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:03.854940 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" event={"ID":"6ea8559a-aa64-4d00-84c0-2c9405e32606","Type":"ContainerStarted","Data":"6631c5126e80f696cd46a7c1d0c462829260a5335f3dec4bf87a1538d1a3445f"} Oct 29 22:42:03.855788 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:03.855257 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" event={"ID":"6ea8559a-aa64-4d00-84c0-2c9405e32606","Type":"ContainerStarted","Data":"18a2a20e51d41dd4a3b60126bc4d6767402d1f5e2366830e11fdc89dc2b36f9d"} Oct 29 22:42:03.855788 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:03.855309 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" event={"ID":"6ea8559a-aa64-4d00-84c0-2c9405e32606","Type":"ContainerStarted","Data":"45333b46b469e8fb2cde1691b144de028bafcf822b89854676f3e931a811ab2f"} Oct 29 22:42:03.855788 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:03.855355 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" event={"ID":"6ea8559a-aa64-4d00-84c0-2c9405e32606","Type":"ContainerStarted","Data":"c5a45a7aa74e6c31ab2daa67ecf06293891ee795fa9daacf457ec40959f364a3"} Oct 29 22:42:03.855788 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:03.855411 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" event={"ID":"6ea8559a-aa64-4d00-84c0-2c9405e32606","Type":"ContainerStarted","Data":"dd6a87627d9752ad619584f48c4bd698dc2ccd0f0577a0deeceacc5ca18b9d74"} Oct 29 22:42:03.855788 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:03.855453 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" event={"ID":"6ea8559a-aa64-4d00-84c0-2c9405e32606","Type":"ContainerStarted","Data":"ba00a0522fcc1f20a79e471d9fe99209817a21b02b9004be34f650234f1a458a"} Oct 29 22:42:03.857743 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:03.857720 2577 generic.go:356] "Generic (PLEG): container finished" podID="c147a7be-d6f6-458e-8b0d-70da7a3c7026" containerID="7db4c610461011a360e45bebc5cd9f3a812debacd4793a26bea16edd32ba0f80" exitCode=0 Oct 29 22:42:03.857826 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:03.857748 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdtdb" event={"ID":"c147a7be-d6f6-458e-8b0d-70da7a3c7026","Type":"ContainerDied","Data":"7db4c610461011a360e45bebc5cd9f3a812debacd4793a26bea16edd32ba0f80"} Oct 29 22:42:04.335390 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:04.335302 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8ad4438-da5a-4f3a-bb5b-07c330aef792-metrics-certs\") pod \"network-metrics-daemon-29dzf\" (UID: \"a8ad4438-da5a-4f3a-bb5b-07c330aef792\") " pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:42:04.335390 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:04.335348 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd35650d-2c53-4230-8518-c4cb05b59558-original-pull-secret\") pod \"global-pull-secret-syncer-ldgnp\" (UID: \"fd35650d-2c53-4230-8518-c4cb05b59558\") " pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:42:04.335546 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:04.335478 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:42:04.335546 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:04.335479 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:42:04.335546 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:04.335545 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd35650d-2c53-4230-8518-c4cb05b59558-original-pull-secret podName:fd35650d-2c53-4230-8518-c4cb05b59558 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:20.335529278 +0000 UTC m=+34.267907605 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fd35650d-2c53-4230-8518-c4cb05b59558-original-pull-secret") pod "global-pull-secret-syncer-ldgnp" (UID: "fd35650d-2c53-4230-8518-c4cb05b59558") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:42:04.335644 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:04.335601 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8ad4438-da5a-4f3a-bb5b-07c330aef792-metrics-certs podName:a8ad4438-da5a-4f3a-bb5b-07c330aef792 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:20.335553062 +0000 UTC m=+34.267931383 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8ad4438-da5a-4f3a-bb5b-07c330aef792-metrics-certs") pod "network-metrics-daemon-29dzf" (UID: "a8ad4438-da5a-4f3a-bb5b-07c330aef792") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:42:04.436528 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:04.436497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6ll2\" (UniqueName: \"kubernetes.io/projected/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff-kube-api-access-f6ll2\") pod \"network-check-target-mbv7x\" (UID: \"b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff\") " pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:42:04.436657 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:04.436614 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:42:04.436657 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:04.436628 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:42:04.436657 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:04.436637 2577 projected.go:196] Error preparing data for projected volume kube-api-access-f6ll2 for pod openshift-network-diagnostics/network-check-target-mbv7x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:42:04.436754 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:04.436681 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff-kube-api-access-f6ll2 podName:b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff nodeName:}" failed. No retries permitted until 2026-10-29 22:42:20.436669566 +0000 UTC m=+34.369047888 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f6ll2" (UniqueName: "kubernetes.io/projected/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff-kube-api-access-f6ll2") pod "network-check-target-mbv7x" (UID: "b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:42:04.559073 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:04.559035 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:42:04.559073 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:04.559060 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:42:04.559295 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:04.559124 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:42:04.559295 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:04.559255 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldgnp" podUID="fd35650d-2c53-4230-8518-c4cb05b59558" Oct 29 22:42:04.559382 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:04.559347 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbv7x" podUID="b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff" Oct 29 22:42:04.559450 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:04.559435 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dzf" podUID="a8ad4438-da5a-4f3a-bb5b-07c330aef792" Oct 29 22:42:04.863068 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:04.863033 2577 generic.go:356] "Generic (PLEG): container finished" podID="c147a7be-d6f6-458e-8b0d-70da7a3c7026" containerID="b8a85846a0be05835d90d7d7d2b12aea5e67e49b1ce0189e1049bae1b6e729b8" exitCode=0 Oct 29 22:42:04.863521 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:04.863097 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdtdb" event={"ID":"c147a7be-d6f6-458e-8b0d-70da7a3c7026","Type":"ContainerDied","Data":"b8a85846a0be05835d90d7d7d2b12aea5e67e49b1ce0189e1049bae1b6e729b8"} Oct 29 22:42:05.868867 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:05.868772 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdtdb" event={"ID":"c147a7be-d6f6-458e-8b0d-70da7a3c7026","Type":"ContainerStarted","Data":"2698c97172b9e8039a65dd7a9598f1fc59a6156326eebefa948968bbfb5a95f6"} Oct 29 22:42:05.871863 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:05.871838 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" event={"ID":"6ea8559a-aa64-4d00-84c0-2c9405e32606","Type":"ContainerStarted","Data":"9d9f769e7f1df5235d8c4bbe927b3552fce0b214ef174b404f744d433b0779e4"} Oct 29 22:42:05.939804 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:05.939746 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rdtdb" podStartSLOduration=8.974136381 podStartE2EDuration="19.939733979s" totalImagesPullingTime="10.965597598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:05.926988429 +0000 UTC m=+19.859366771" watchObservedRunningTime="2026-10-29 22:42:05.939733979 +0000 UTC m=+19.872112321" Oct 29 22:42:06.017851 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:06.017821 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jzjsw" Oct 29 22:42:06.018003 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:06.017938 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:42:06.018488 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:06.018461 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jzjsw" Oct 29 22:42:06.559335 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:06.559302 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:42:06.559510 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:06.559445 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:42:06.559510 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:06.559458 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldgnp" podUID="fd35650d-2c53-4230-8518-c4cb05b59558" Oct 29 22:42:06.559645 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:06.559574 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:42:06.559700 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:06.559564 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbv7x" podUID="b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff" Oct 29 22:42:06.559700 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:06.559674 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dzf" podUID="a8ad4438-da5a-4f3a-bb5b-07c330aef792" Oct 29 22:42:07.880049 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:07.879864 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" event={"ID":"6ea8559a-aa64-4d00-84c0-2c9405e32606","Type":"ContainerStarted","Data":"2040420a5b4d568fdcb2b962bcf4be76cc4a378d25558de764fe3f0c52280b57"} Oct 29 22:42:07.880556 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:07.880102 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:07.880556 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:07.880156 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:07.896147 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:07.896122 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:07.970101 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:07.970063 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" podStartSLOduration=7.816079887 podStartE2EDuration="21.970040795s" totalImagesPullingTime="14.153960908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:07.969965269 +0000 UTC m=+21.902343611" watchObservedRunningTime="2026-10-29 22:42:07.970040795 +0000 UTC m=+21.902419138" Oct 29 22:42:08.558320 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:08.558287 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:42:08.558320 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:08.558312 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:42:08.558621 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:08.558413 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbv7x" podUID="b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff" Oct 29 22:42:08.558621 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:08.558426 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:42:08.558621 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:08.558506 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldgnp" podUID="fd35650d-2c53-4230-8518-c4cb05b59558" Oct 29 22:42:08.558621 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:08.558608 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dzf" podUID="a8ad4438-da5a-4f3a-bb5b-07c330aef792" Oct 29 22:42:08.882424 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:08.882395 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:08.897543 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:08.897519 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:09.667770 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:09.667696 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mbv7x"] Oct 29 22:42:09.667889 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:09.667792 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:42:09.667889 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:09.667868 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbv7x" podUID="b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff" Oct 29 22:42:09.672318 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:09.672294 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ldgnp"] Oct 29 22:42:09.672444 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:09.672375 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:42:09.672492 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:09.672475 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldgnp" podUID="fd35650d-2c53-4230-8518-c4cb05b59558" Oct 29 22:42:09.672854 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:09.672837 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-29dzf"] Oct 29 22:42:09.672931 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:09.672920 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:42:09.673016 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:09.672998 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dzf" podUID="a8ad4438-da5a-4f3a-bb5b-07c330aef792" Oct 29 22:42:11.559084 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:11.559039 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:42:11.559084 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:11.559087 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:42:11.559542 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:11.559066 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:42:11.559542 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:11.559150 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbv7x" podUID="b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff" Oct 29 22:42:11.559542 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:11.559258 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dzf" podUID="a8ad4438-da5a-4f3a-bb5b-07c330aef792" Oct 29 22:42:11.559542 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:11.559321 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldgnp" podUID="fd35650d-2c53-4230-8518-c4cb05b59558" Oct 29 22:42:13.558258 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:13.558224 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:42:13.558686 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:13.558234 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:42:13.558686 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:13.558331 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dzf" podUID="a8ad4438-da5a-4f3a-bb5b-07c330aef792" Oct 29 22:42:13.558686 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:13.558356 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:42:13.558686 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:13.558438 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ldgnp" podUID="fd35650d-2c53-4230-8518-c4cb05b59558" Oct 29 22:42:13.558686 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:13.558527 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mbv7x" podUID="b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff" Oct 29 22:42:14.834492 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:14.834321 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeReady" Oct 29 22:42:14.834837 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:14.834594 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 29 22:42:14.894869 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:14.894835 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9g7rk"] Oct 29 22:42:14.898558 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:14.898495 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-k7c6q"] Oct 29 22:42:14.898664 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:14.898620 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9g7rk" Oct 29 22:42:14.903984 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:14.903952 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:14.911640 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:14.911622 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zb2ch"] Oct 29 22:42:14.916354 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:14.916334 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:14.919580 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:14.919558 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 29 22:42:14.919963 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:14.919754 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 29 22:42:14.919963 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:14.919754 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xzwzj\"" Oct 29 22:42:14.919963 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:14.919828 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 29 22:42:14.920500 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:14.920483 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9g7rk"] Oct 29 22:42:14.923309 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:14.923291 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 29 22:42:14.923515 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:14.923503 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 29 22:42:14.929569 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:14.929547 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k7c6q"] Oct 29 22:42:14.936918 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:14.936898 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zb2ch"] Oct 29 22:42:15.015265 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.015242 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae4b768-8a6b-4c10-ae02-c1a3950d830c-tmp-dir\") pod \"dns-default-zb2ch\" (UID: \"1ae4b768-8a6b-4c10-ae02-c1a3950d830c\") " pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:15.015380 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.015284 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dab4a141-b01a-4434-a792-d8fc9d63a83d-cert\") pod \"ingress-canary-9g7rk\" (UID: \"dab4a141-b01a-4434-a792-d8fc9d63a83d\") " pod="openshift-ingress-canary/ingress-canary-9g7rk" Oct 29 22:42:15.015380 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.015303 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ae4b768-8a6b-4c10-ae02-c1a3950d830c-metrics-tls\") pod \"dns-default-zb2ch\" (UID: \"1ae4b768-8a6b-4c10-ae02-c1a3950d830c\") " pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:15.015380 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.015361 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae4b768-8a6b-4c10-ae02-c1a3950d830c-config-volume\") pod \"dns-default-zb2ch\" (UID: \"1ae4b768-8a6b-4c10-ae02-c1a3950d830c\") " pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:15.015478 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.015406 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7tgw\" (UniqueName: \"kubernetes.io/projected/dab4a141-b01a-4434-a792-d8fc9d63a83d-kube-api-access-k7tgw\") pod \"ingress-canary-9g7rk\" (UID: \"dab4a141-b01a-4434-a792-d8fc9d63a83d\") " pod="openshift-ingress-canary/ingress-canary-9g7rk" Oct 29 22:42:15.015478 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.015461 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4026e493-67d9-4b2c-8267-1da3a0b1d621-data-volume\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:15.015543 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.015479 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhdc6\" (UniqueName: \"kubernetes.io/projected/1ae4b768-8a6b-4c10-ae02-c1a3950d830c-kube-api-access-hhdc6\") pod \"dns-default-zb2ch\" (UID: \"1ae4b768-8a6b-4c10-ae02-c1a3950d830c\") " pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:15.015543 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.015496 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4026e493-67d9-4b2c-8267-1da3a0b1d621-crio-socket\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:15.015543 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.015513 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4026e493-67d9-4b2c-8267-1da3a0b1d621-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:15.015627 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.015596 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hplpk\" (UniqueName: \"kubernetes.io/projected/4026e493-67d9-4b2c-8267-1da3a0b1d621-kube-api-access-hplpk\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:15.015658 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.015626 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4026e493-67d9-4b2c-8267-1da3a0b1d621-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:15.019780 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.019758 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 29 22:42:15.019912 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.019895 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 29 22:42:15.019967 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.019939 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 29 22:42:15.019967 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.019948 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 29 22:42:15.019967 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.019962 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c4xfj\"" Oct 29 22:42:15.020140 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.020127 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gg4pb\"" Oct 29 22:42:15.116741 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.116711 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4026e493-67d9-4b2c-8267-1da3a0b1d621-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:15.116890 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.116747 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae4b768-8a6b-4c10-ae02-c1a3950d830c-tmp-dir\") pod \"dns-default-zb2ch\" (UID: \"1ae4b768-8a6b-4c10-ae02-c1a3950d830c\") " pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:15.116890 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.116773 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dab4a141-b01a-4434-a792-d8fc9d63a83d-cert\") pod \"ingress-canary-9g7rk\" (UID: \"dab4a141-b01a-4434-a792-d8fc9d63a83d\") " pod="openshift-ingress-canary/ingress-canary-9g7rk" Oct 29 22:42:15.116890 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.116801 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ae4b768-8a6b-4c10-ae02-c1a3950d830c-metrics-tls\") pod \"dns-default-zb2ch\" (UID: \"1ae4b768-8a6b-4c10-ae02-c1a3950d830c\") " pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:15.116890 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.116818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae4b768-8a6b-4c10-ae02-c1a3950d830c-config-volume\") pod \"dns-default-zb2ch\" (UID: \"1ae4b768-8a6b-4c10-ae02-c1a3950d830c\") " pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:15.116890 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.116839 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7tgw\" (UniqueName: \"kubernetes.io/projected/dab4a141-b01a-4434-a792-d8fc9d63a83d-kube-api-access-k7tgw\") pod \"ingress-canary-9g7rk\" (UID: \"dab4a141-b01a-4434-a792-d8fc9d63a83d\") " pod="openshift-ingress-canary/ingress-canary-9g7rk" Oct 29 22:42:15.116890 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.116861 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4026e493-67d9-4b2c-8267-1da3a0b1d621-data-volume\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:15.116890 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.116883 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhdc6\" (UniqueName: \"kubernetes.io/projected/1ae4b768-8a6b-4c10-ae02-c1a3950d830c-kube-api-access-hhdc6\") pod \"dns-default-zb2ch\" (UID: \"1ae4b768-8a6b-4c10-ae02-c1a3950d830c\") " pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:15.117272 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.116908 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4026e493-67d9-4b2c-8267-1da3a0b1d621-crio-socket\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:15.117272 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.116935 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4026e493-67d9-4b2c-8267-1da3a0b1d621-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:15.117272 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.116981 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hplpk\" (UniqueName: \"kubernetes.io/projected/4026e493-67d9-4b2c-8267-1da3a0b1d621-kube-api-access-hplpk\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:15.117272 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.117136 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae4b768-8a6b-4c10-ae02-c1a3950d830c-tmp-dir\") pod \"dns-default-zb2ch\" (UID: \"1ae4b768-8a6b-4c10-ae02-c1a3950d830c\") " pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:15.117272 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.117184 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4026e493-67d9-4b2c-8267-1da3a0b1d621-data-volume\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:15.117523 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.117303 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4026e493-67d9-4b2c-8267-1da3a0b1d621-crio-socket\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:15.117574 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.117523 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae4b768-8a6b-4c10-ae02-c1a3950d830c-config-volume\") pod \"dns-default-zb2ch\" (UID: \"1ae4b768-8a6b-4c10-ae02-c1a3950d830c\") " pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:15.117714 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.117686 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4026e493-67d9-4b2c-8267-1da3a0b1d621-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:15.121014 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.120993 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4026e493-67d9-4b2c-8267-1da3a0b1d621-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:15.121113 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.121041 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ae4b768-8a6b-4c10-ae02-c1a3950d830c-metrics-tls\") pod \"dns-default-zb2ch\" (UID: \"1ae4b768-8a6b-4c10-ae02-c1a3950d830c\") " pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:15.121113 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.121083 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dab4a141-b01a-4434-a792-d8fc9d63a83d-cert\") pod \"ingress-canary-9g7rk\" (UID: \"dab4a141-b01a-4434-a792-d8fc9d63a83d\") " pod="openshift-ingress-canary/ingress-canary-9g7rk" Oct 29 22:42:15.140581 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.140551 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hplpk\" (UniqueName: \"kubernetes.io/projected/4026e493-67d9-4b2c-8267-1da3a0b1d621-kube-api-access-hplpk\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:15.140697 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.140613 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhdc6\" (UniqueName: \"kubernetes.io/projected/1ae4b768-8a6b-4c10-ae02-c1a3950d830c-kube-api-access-hhdc6\") pod \"dns-default-zb2ch\" (UID: \"1ae4b768-8a6b-4c10-ae02-c1a3950d830c\") " pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:15.140697 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.140644 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7tgw\" (UniqueName: \"kubernetes.io/projected/dab4a141-b01a-4434-a792-d8fc9d63a83d-kube-api-access-k7tgw\") pod \"ingress-canary-9g7rk\" (UID: \"dab4a141-b01a-4434-a792-d8fc9d63a83d\") " pod="openshift-ingress-canary/ingress-canary-9g7rk" Oct 29 22:42:15.215782 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.215702 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9g7rk" Oct 29 22:42:15.227019 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.226988 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:15.232593 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.232564 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:15.558336 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.558253 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:42:15.558336 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.558264 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:42:15.558544 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.558398 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:42:15.569584 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.569564 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9g7rk"] Oct 29 22:42:15.572999 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.572928 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zb2ch"] Oct 29 22:42:15.573905 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.573885 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rgvrg\"" Oct 29 22:42:15.574135 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:42:15.574102 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab4a141_b01a_4434_a792_d8fc9d63a83d.slice/crio-1135b6d0397687f9487e54cfc133c8d6a93b6b9cccf91599c4b75ef7c3cddf85 WatchSource:0}: Error finding container 1135b6d0397687f9487e54cfc133c8d6a93b6b9cccf91599c4b75ef7c3cddf85: Status 404 returned error can't find the container with id 1135b6d0397687f9487e54cfc133c8d6a93b6b9cccf91599c4b75ef7c3cddf85 Oct 29 22:42:15.574217 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.574170 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k7c6q"] Oct 29 22:42:15.577011 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:42:15.576926 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae4b768_8a6b_4c10_ae02_c1a3950d830c.slice/crio-f814ce24c2e1df862efd858815d8f17a65493d2b023fa5bdce3352d728002867 WatchSource:0}: Error finding container f814ce24c2e1df862efd858815d8f17a65493d2b023fa5bdce3352d728002867: Status 404 returned error can't find the container with id f814ce24c2e1df862efd858815d8f17a65493d2b023fa5bdce3352d728002867 Oct 29 22:42:15.577617 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:42:15.577593 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4026e493_67d9_4b2c_8267_1da3a0b1d621.slice/crio-59ab3ed9c9fb828acdc99da907d98147baa0b8793345fd3d6c9f8e932ef21779 WatchSource:0}: Error finding container 59ab3ed9c9fb828acdc99da907d98147baa0b8793345fd3d6c9f8e932ef21779: Status 404 returned error can't find the container with id 59ab3ed9c9fb828acdc99da907d98147baa0b8793345fd3d6c9f8e932ef21779 Oct 29 22:42:15.580795 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.580776 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 22:42:15.580861 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.580811 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 22:42:15.581090 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.581076 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 22:42:15.581466 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.581370 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 22:42:15.584239 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.584219 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-dvjrv\"" Oct 29 22:42:15.584239 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.584232 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zrmx7\"" Oct 29 22:42:15.896994 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.896802 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9g7rk" event={"ID":"dab4a141-b01a-4434-a792-d8fc9d63a83d","Type":"ContainerStarted","Data":"1135b6d0397687f9487e54cfc133c8d6a93b6b9cccf91599c4b75ef7c3cddf85"} Oct 29 22:42:15.897924 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.897900 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zb2ch" event={"ID":"1ae4b768-8a6b-4c10-ae02-c1a3950d830c","Type":"ContainerStarted","Data":"f814ce24c2e1df862efd858815d8f17a65493d2b023fa5bdce3352d728002867"} Oct 29 22:42:15.899258 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.899236 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k7c6q" event={"ID":"4026e493-67d9-4b2c-8267-1da3a0b1d621","Type":"ContainerStarted","Data":"b43219052c6efb79afab2fed9526a46a0038b1a8a994105f9088a2c6b2b5288d"} Oct 29 22:42:15.899367 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:15.899276 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k7c6q" event={"ID":"4026e493-67d9-4b2c-8267-1da3a0b1d621","Type":"ContainerStarted","Data":"59ab3ed9c9fb828acdc99da907d98147baa0b8793345fd3d6c9f8e932ef21779"} Oct 29 22:42:16.905586 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:16.905539 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k7c6q" event={"ID":"4026e493-67d9-4b2c-8267-1da3a0b1d621","Type":"ContainerStarted","Data":"a8f48f0de4a671b3df42572a30610cd8140566eae6ece2be8eefe7d2a494842a"} Oct 29 22:42:17.910455 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:17.910374 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zb2ch" event={"ID":"1ae4b768-8a6b-4c10-ae02-c1a3950d830c","Type":"ContainerStarted","Data":"cb8befd66ac625736f597c44bf08876dc564089a39e88ec86dbb3599871eefe2"} Oct 29 22:42:17.910455 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:17.910439 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zb2ch" event={"ID":"1ae4b768-8a6b-4c10-ae02-c1a3950d830c","Type":"ContainerStarted","Data":"1b921bb002d72b116ac49eb5515a2007b3038f5e8b6561e9a4c0ef4f2486235e"} Oct 29 22:42:17.910933 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:17.910513 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:17.953818 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:17.953767 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-zb2ch" podStartSLOduration=1.91669479 podStartE2EDuration="3.953752988s" totalImagesPullingTime="2.037058198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:17.952801254 +0000 UTC m=+31.885179599" watchObservedRunningTime="2026-10-29 22:42:17.953752988 +0000 UTC m=+31.886131360" Oct 29 22:42:18.915268 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:18.915227 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9g7rk" event={"ID":"dab4a141-b01a-4434-a792-d8fc9d63a83d","Type":"ContainerStarted","Data":"7249c39e3541dd0f7dfe01f034cade3f1f959640cbb8a9e7db1352497abae7f0"} Oct 29 22:42:18.917377 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:18.917350 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k7c6q" event={"ID":"4026e493-67d9-4b2c-8267-1da3a0b1d621","Type":"ContainerStarted","Data":"5633a930d09ef4c5aadbb358b474780803059af062b1ef43837c3b97e7ab8f12"} Oct 29 22:42:18.957335 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:18.957258 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9g7rk" podStartSLOduration=1.851785414 podStartE2EDuration="4.957246323s" totalImagesPullingTime="3.105460909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:18.956173805 +0000 UTC m=+32.888552149" watchObservedRunningTime="2026-10-29 22:42:18.957246323 +0000 UTC m=+32.889624658" Oct 29 22:42:19.000535 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:19.000481 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-k7c6q" podStartSLOduration=2.624660132 podStartE2EDuration="5.000463427s" totalImagesPullingTime="2.375803295s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:19.000042486 +0000 UTC m=+32.932420850" watchObservedRunningTime="2026-10-29 22:42:19.000463427 +0000 UTC m=+32.932841775" Oct 29 22:42:20.364784 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:20.364744 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd35650d-2c53-4230-8518-c4cb05b59558-original-pull-secret\") pod \"global-pull-secret-syncer-ldgnp\" (UID: \"fd35650d-2c53-4230-8518-c4cb05b59558\") " pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:42:20.365339 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:20.364834 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8ad4438-da5a-4f3a-bb5b-07c330aef792-metrics-certs\") pod \"network-metrics-daemon-29dzf\" (UID: \"a8ad4438-da5a-4f3a-bb5b-07c330aef792\") " pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:42:20.368024 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:20.367997 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd35650d-2c53-4230-8518-c4cb05b59558-original-pull-secret\") pod \"global-pull-secret-syncer-ldgnp\" (UID: \"fd35650d-2c53-4230-8518-c4cb05b59558\") " pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:42:20.368119 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:20.368005 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8ad4438-da5a-4f3a-bb5b-07c330aef792-metrics-certs\") pod \"network-metrics-daemon-29dzf\" (UID: \"a8ad4438-da5a-4f3a-bb5b-07c330aef792\") " pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:42:20.372910 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:20.372889 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:42:20.388212 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:20.388170 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:42:20.465890 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:20.465861 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6ll2\" (UniqueName: \"kubernetes.io/projected/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff-kube-api-access-f6ll2\") pod \"network-check-target-mbv7x\" (UID: \"b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff\") " pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:42:20.468308 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:20.468285 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6ll2\" (UniqueName: \"kubernetes.io/projected/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff-kube-api-access-f6ll2\") pod \"network-check-target-mbv7x\" (UID: \"b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff\") " pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:42:20.680859 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:20.680778 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:42:20.747656 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:20.747620 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ldgnp"] Oct 29 22:42:20.750667 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:42:20.750637 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd35650d_2c53_4230_8518_c4cb05b59558.slice/crio-addbe92aa928a46f8b9299d8e767868b40ac40aa9b24142658573912c660c257 WatchSource:0}: Error finding container addbe92aa928a46f8b9299d8e767868b40ac40aa9b24142658573912c660c257: Status 404 returned error can't find the container with id addbe92aa928a46f8b9299d8e767868b40ac40aa9b24142658573912c660c257 Oct 29 22:42:20.773976 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:20.773948 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-29dzf"] Oct 29 22:42:20.777000 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:42:20.776964 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ad4438_da5a_4f3a_bb5b_07c330aef792.slice/crio-16891bc02d310862d1fc552da31b7cc0205b945c4d2f2479f1b95eafdbaed289 WatchSource:0}: Error finding container 16891bc02d310862d1fc552da31b7cc0205b945c4d2f2479f1b95eafdbaed289: Status 404 returned error can't find the container with id 16891bc02d310862d1fc552da31b7cc0205b945c4d2f2479f1b95eafdbaed289 Oct 29 22:42:20.925254 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:20.925217 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-29dzf" event={"ID":"a8ad4438-da5a-4f3a-bb5b-07c330aef792","Type":"ContainerStarted","Data":"16891bc02d310862d1fc552da31b7cc0205b945c4d2f2479f1b95eafdbaed289"} Oct 29 22:42:20.926374 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:20.926342 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ldgnp" event={"ID":"fd35650d-2c53-4230-8518-c4cb05b59558","Type":"ContainerStarted","Data":"addbe92aa928a46f8b9299d8e767868b40ac40aa9b24142658573912c660c257"} Oct 29 22:42:21.007518 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:21.007447 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mbv7x"] Oct 29 22:42:21.010241 ip-10-0-130-252 kubenswrapper[2577]: W1029 22:42:21.010218 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f1f5a0_d3e4_4876_8a15_68fe2ff0ddff.slice/crio-9597c7254279a4acc36b251def7dec3cb05d5ecffc4b49fe9fc95a4bcdeb70c7 WatchSource:0}: Error finding container 9597c7254279a4acc36b251def7dec3cb05d5ecffc4b49fe9fc95a4bcdeb70c7: Status 404 returned error can't find the container with id 9597c7254279a4acc36b251def7dec3cb05d5ecffc4b49fe9fc95a4bcdeb70c7 Oct 29 22:42:21.931640 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:21.931595 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mbv7x" event={"ID":"b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff","Type":"ContainerStarted","Data":"9597c7254279a4acc36b251def7dec3cb05d5ecffc4b49fe9fc95a4bcdeb70c7"} Oct 29 22:42:22.938630 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:22.938593 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-29dzf" event={"ID":"a8ad4438-da5a-4f3a-bb5b-07c330aef792","Type":"ContainerStarted","Data":"1b7899ac653dad7b2e192781224491c66c095a648788757d3624f5a9c232766a"} Oct 29 22:42:22.939041 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:22.938645 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-29dzf" event={"ID":"a8ad4438-da5a-4f3a-bb5b-07c330aef792","Type":"ContainerStarted","Data":"b3031ef356ce18c093ec0ae5c5b5c18d22844e604b5dce967c1a50fb9cdd0985"} Oct 29 22:42:22.980809 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:22.980743 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-29dzf" podStartSLOduration=35.506425616 podStartE2EDuration="36.980726761s" totalImagesPullingTime="1.474301145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:22.980369856 +0000 UTC m=+36.912748196" watchObservedRunningTime="2026-10-29 22:42:22.980726761 +0000 UTC m=+36.913105103" Oct 29 22:42:24.390537 ip-10-0-130-252 kubenswrapper[2577]: E1029 22:42:24.390499 2577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:42:24.946740 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:24.946705 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mbv7x" event={"ID":"b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff","Type":"ContainerStarted","Data":"484c3c16fddd9477dce447f76a22b9e1cbf08a8400a80f28e00ca237ec33c621"} Oct 29 22:42:24.946917 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:24.946851 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:42:24.988128 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:24.987026 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mbv7x" podStartSLOduration=34.486054498 podStartE2EDuration="37.987006273s" totalImagesPullingTime="3.500951775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:24.986664001 +0000 UTC m=+38.919042440" watchObservedRunningTime="2026-10-29 22:42:24.987006273 +0000 UTC m=+38.919384646" Oct 29 22:42:25.952539 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:25.952442 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ldgnp" event={"ID":"fd35650d-2c53-4230-8518-c4cb05b59558","Type":"ContainerStarted","Data":"19457b04e8c3d74889ed4497d93c34722f51e25a1f0e9486bf797d6993c5c1c5"} Oct 29 22:42:25.989304 ip-10-0-130-252 kubenswrapper[2577]: I1029 22:42:25.989173 2577 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:42:25.989328 ip-10-0-130-252 systemd[1]: Stopping Kubernetes Kubelet... Oct 29 22:42:25.997596 ip-10-0-130-252 systemd[1]: kubelet.service: Deactivated successfully. Oct 29 22:42:25.997771 ip-10-0-130-252 systemd[1]: Stopped Kubernetes Kubelet. Oct 29 22:42:25.997947 ip-10-0-130-252 systemd[1]: kubelet.service: Consumed 2.532s CPU time. Oct 29 22:42:26.010664 ip-10-0-130-252 systemd[1]: Starting Kubernetes Kubelet... Oct 29 22:42:26.089515 ip-10-0-130-252 kubenswrapper[4821]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:42:26.089515 ip-10-0-130-252 kubenswrapper[4821]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 29 22:42:26.089515 ip-10-0-130-252 kubenswrapper[4821]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:42:26.089515 ip-10-0-130-252 kubenswrapper[4821]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:42:26.091708 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091622 4821 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:42:26.091708 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091706 4821 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:42:26.091708 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091711 4821 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:42:26.091708 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091714 4821 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:42:26.091862 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091717 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:42:26.091862 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091720 4821 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:42:26.091862 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091723 4821 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:42:26.091862 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091726 4821 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:42:26.091862 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091729 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:42:26.091862 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091732 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:42:26.091862 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091735 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:42:26.091862 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091737 4821 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:42:26.091862 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091740 4821 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:42:26.091862 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091743 4821 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:42:26.091862 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091746 4821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:42:26.091862 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091748 4821 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:42:26.091862 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091751 4821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:42:26.091862 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091754 4821 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:42:26.091862 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091757 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:42:26.091862 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091759 4821 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:42:26.091862 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091762 4821 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:42:26.091862 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091769 4821 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:42:26.091862 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091772 4821 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:42:26.092394 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091777 4821 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:42:26.092394 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091781 4821 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:42:26.092394 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091783 4821 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:42:26.092394 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091786 4821 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:42:26.092394 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091789 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:42:26.092394 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091791 4821 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:42:26.092394 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091794 4821 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:42:26.092394 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091797 4821 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:42:26.092394 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091799 4821 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:42:26.092394 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091802 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:42:26.092394 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091805 4821 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:42:26.092394 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091809 4821 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:42:26.092394 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091812 4821 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:42:26.092394 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091816 4821 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:42:26.092394 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091819 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:42:26.092394 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091822 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:42:26.092394 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091830 4821 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:42:26.092394 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091838 4821 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:42:26.092394 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091841 4821 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:42:26.092850 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091844 4821 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:42:26.092850 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091846 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:42:26.092850 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091849 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:42:26.092850 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091852 4821 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:42:26.092850 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091854 4821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:42:26.092850 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091857 4821 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:42:26.092850 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091859 4821 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:42:26.092850 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091862 4821 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:42:26.092850 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091864 4821 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:42:26.092850 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091867 4821 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:42:26.092850 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091869 4821 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:42:26.092850 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091872 4821 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:42:26.092850 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091875 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:42:26.092850 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091878 4821 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:42:26.092850 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091880 4821 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:42:26.092850 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091883 4821 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:42:26.092850 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091885 4821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:42:26.092850 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091888 4821 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:42:26.092850 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091890 4821 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:42:26.093337 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091893 4821 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:42:26.093337 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091896 4821 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:42:26.093337 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091898 4821 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:42:26.093337 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091900 4821 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:42:26.093337 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091903 4821 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:42:26.093337 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091906 4821 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:42:26.093337 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091908 4821 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:42:26.093337 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091910 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:42:26.093337 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091913 4821 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:42:26.093337 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091916 4821 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:42:26.093337 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091918 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:42:26.093337 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091928 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:42:26.093337 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091932 4821 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:42:26.093337 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091935 4821 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:42:26.093337 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091937 4821 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:42:26.093337 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091940 4821 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:42:26.093337 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091942 4821 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:42:26.093337 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091945 4821 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:42:26.093337 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091954 4821 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:42:26.093337 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091957 4821 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:42:26.093820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091960 4821 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:42:26.093820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091962 4821 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:42:26.093820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091965 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:42:26.093820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091968 4821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:42:26.093820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091970 4821 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:42:26.093820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091973 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:42:26.093820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091975 4821 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:42:26.093820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091978 4821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:42:26.093820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091980 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:42:26.093820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091983 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:42:26.093820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091985 4821 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:42:26.093820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091988 4821 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:42:26.093820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091991 4821 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:42:26.093820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091993 4821 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:42:26.093820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091995 4821 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:42:26.093820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.091998 4821 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:42:26.093820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092001 4821 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:42:26.093820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092003 4821 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:42:26.093820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092006 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:42:26.094330 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092008 4821 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:42:26.094330 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092011 4821 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:42:26.094330 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092015 4821 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:42:26.094330 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092017 4821 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:42:26.094330 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092021 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:42:26.094330 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092024 4821 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:42:26.094330 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092819 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:42:26.094330 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092826 4821 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:42:26.094330 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092829 4821 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:42:26.094330 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092832 4821 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:42:26.094330 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092835 4821 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:42:26.094330 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092837 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:42:26.094330 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092840 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:42:26.094330 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092842 4821 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:42:26.094330 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092845 4821 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:42:26.094330 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092848 4821 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:42:26.094330 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092851 4821 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:42:26.094330 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092854 4821 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:42:26.094330 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092856 4821 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:42:26.094330 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092859 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:42:26.094827 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092862 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:42:26.094827 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092864 4821 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:42:26.094827 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092867 4821 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:42:26.094827 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092870 4821 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:42:26.094827 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092872 4821 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:42:26.094827 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092875 4821 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:42:26.094827 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092877 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:42:26.094827 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092880 4821 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:42:26.094827 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092882 4821 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:42:26.094827 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092885 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:42:26.094827 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092887 4821 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:42:26.094827 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092890 4821 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:42:26.094827 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092893 4821 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:42:26.094827 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092895 4821 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:42:26.094827 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092899 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:42:26.094827 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092901 4821 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:42:26.094827 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092906 4821 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:42:26.094827 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092908 4821 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:42:26.094827 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092912 4821 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:42:26.094827 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092915 4821 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:42:26.095336 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092917 4821 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:42:26.095336 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092920 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:42:26.095336 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092922 4821 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:42:26.095336 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092925 4821 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:42:26.095336 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092927 4821 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:42:26.095336 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092930 4821 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:42:26.095336 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092932 4821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:42:26.095336 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092935 4821 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:42:26.095336 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092938 4821 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:42:26.095336 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092941 4821 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:42:26.095336 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092943 4821 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:42:26.095336 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092946 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:42:26.095336 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092948 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:42:26.095336 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092951 4821 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:42:26.095336 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092953 4821 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:42:26.095336 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092956 4821 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:42:26.095336 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092958 4821 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:42:26.095336 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092960 4821 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:42:26.095336 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092963 4821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:42:26.095814 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092965 4821 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:42:26.095814 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092968 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:42:26.095814 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092970 4821 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:42:26.095814 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092973 4821 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:42:26.095814 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092975 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:42:26.095814 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092977 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:42:26.095814 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092980 4821 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:42:26.095814 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092983 4821 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:42:26.095814 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092985 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:42:26.095814 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092989 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:42:26.095814 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092992 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:42:26.095814 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092994 4821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:42:26.095814 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.092998 4821 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:42:26.095814 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093000 4821 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:42:26.095814 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093005 4821 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:42:26.095814 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093008 4821 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:42:26.095814 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093011 4821 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:42:26.095814 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093013 4821 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:42:26.095814 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093015 4821 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:42:26.096307 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093018 4821 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:42:26.096307 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093020 4821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:42:26.096307 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093024 4821 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:42:26.096307 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093026 4821 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:42:26.096307 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093029 4821 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:42:26.096307 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093031 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:42:26.096307 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093034 4821 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:42:26.096307 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093036 4821 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:42:26.096307 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093038 4821 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:42:26.096307 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093041 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:42:26.096307 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093044 4821 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:42:26.096307 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093047 4821 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:42:26.096307 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093050 4821 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:42:26.096307 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093052 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:42:26.096307 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093055 4821 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:42:26.096307 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093059 4821 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:42:26.096307 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093062 4821 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:42:26.096307 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093065 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:42:26.096307 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093067 4821 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:42:26.096307 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093070 4821 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:42:26.096797 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093072 4821 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:42:26.096797 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093075 4821 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:42:26.096797 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093079 4821 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:42:26.096797 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093081 4821 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:42:26.096797 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093084 4821 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:42:26.096797 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093086 4821 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:42:26.096797 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093089 4821 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:42:26.096797 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093092 4821 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:42:26.096797 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093094 4821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:42:26.096797 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093097 4821 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:42:26.096797 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093099 4821 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:42:26.096797 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093102 4821 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:42:26.096797 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093106 4821 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:42:26.096797 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093110 4821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:42:26.096797 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093646 4821 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:42:26.096797 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093654 4821 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:42:26.096797 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093657 4821 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:42:26.096797 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093660 4821 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:42:26.096797 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093664 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:42:26.096797 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093666 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:42:26.097311 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093669 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:42:26.097311 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093672 4821 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:42:26.097311 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093674 4821 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:42:26.097311 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093677 4821 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:42:26.097311 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093680 4821 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:42:26.097311 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093682 4821 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:42:26.097311 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093685 4821 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:42:26.097311 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093688 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:42:26.097311 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093691 4821 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:42:26.097311 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093693 4821 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:42:26.097311 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093696 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:42:26.097311 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093699 4821 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:42:26.097311 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093701 4821 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:42:26.097311 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093704 4821 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:42:26.097311 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093709 4821 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:42:26.097311 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093712 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:42:26.097311 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093714 4821 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:42:26.097311 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093717 4821 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:42:26.097311 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093720 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:42:26.097948 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093722 4821 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:42:26.097948 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093725 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:42:26.097948 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093728 4821 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:42:26.097948 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093730 4821 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:42:26.097948 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093733 4821 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:42:26.097948 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093735 4821 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:42:26.097948 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093737 4821 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:42:26.097948 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093740 4821 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:42:26.097948 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093742 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:42:26.097948 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093745 4821 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:42:26.097948 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093747 4821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:42:26.097948 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093750 4821 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:42:26.097948 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093753 4821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:42:26.097948 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093755 4821 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:42:26.097948 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093757 4821 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:42:26.097948 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093760 4821 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:42:26.097948 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093762 4821 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:42:26.097948 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093765 4821 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:42:26.097948 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093767 4821 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:42:26.097948 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093770 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:42:26.098726 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093772 4821 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:42:26.098726 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093775 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:42:26.098726 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093777 4821 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:42:26.098726 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093780 4821 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:42:26.098726 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093783 4821 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:42:26.098726 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093785 4821 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:42:26.098726 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093789 4821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:42:26.098726 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093795 4821 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:42:26.098726 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093799 4821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:42:26.098726 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093802 4821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:42:26.098726 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093805 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:42:26.098726 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093808 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:42:26.098726 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093810 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:42:26.098726 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093813 4821 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:42:26.098726 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093816 4821 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:42:26.098726 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093818 4821 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:42:26.098726 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093820 4821 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:42:26.098726 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093823 4821 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:42:26.098726 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093826 4821 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:42:26.099359 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093829 4821 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:42:26.099359 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093832 4821 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:42:26.099359 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093834 4821 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:42:26.099359 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093837 4821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:42:26.099359 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093841 4821 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:42:26.099359 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093845 4821 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:42:26.099359 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093847 4821 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:42:26.099359 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093850 4821 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:42:26.099359 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093854 4821 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:42:26.099359 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093856 4821 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:42:26.099359 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093859 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:42:26.099359 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093861 4821 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:42:26.099359 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093864 4821 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:42:26.099359 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093866 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:42:26.099359 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093869 4821 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:42:26.099359 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093871 4821 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:42:26.099359 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093874 4821 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:42:26.099359 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093876 4821 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:42:26.099359 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093879 4821 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:42:26.099359 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093883 4821 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:42:26.099912 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093888 4821 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:42:26.099912 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093891 4821 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:42:26.099912 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093893 4821 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:42:26.099912 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093896 4821 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:42:26.099912 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093898 4821 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:42:26.099912 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093901 4821 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:42:26.099912 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093904 4821 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:42:26.099912 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093907 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:42:26.099912 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093909 4821 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:42:26.099912 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093912 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:42:26.099912 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093914 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:42:26.099912 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093917 4821 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:42:26.099912 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093919 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:42:26.099912 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093921 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:42:26.099912 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093924 4821 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:42:26.099912 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093926 4821 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:42:26.099912 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093929 4821 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:42:26.099912 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093931 4821 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:42:26.099912 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093934 4821 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:42:26.099912 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093937 4821 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:42:26.100426 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093939 4821 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:42:26.100426 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.093942 4821 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:42:26.100426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094038 4821 flags.go:64] FLAG: --address="0.0.0.0" Oct 29 22:42:26.100426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094045 4821 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 29 22:42:26.100426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094054 4821 flags.go:64] FLAG: --anonymous-auth="true" Oct 29 22:42:26.100426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094060 4821 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 29 22:42:26.100426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094064 4821 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 29 22:42:26.100426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094067 4821 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 29 22:42:26.100426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094074 4821 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 29 22:42:26.100426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094078 4821 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 29 22:42:26.100426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094081 4821 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 29 22:42:26.100426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094086 4821 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 29 22:42:26.100426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094090 4821 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 29 22:42:26.100426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094094 4821 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 29 22:42:26.100426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094097 4821 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 29 22:42:26.100426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094100 4821 flags.go:64] FLAG: --cgroup-root="" Oct 29 22:42:26.100426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094103 4821 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 29 22:42:26.100426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094106 4821 flags.go:64] FLAG: --client-ca-file="" Oct 29 22:42:26.100426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094109 4821 flags.go:64] FLAG: --cloud-provider="external" Oct 29 22:42:26.100426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094113 4821 flags.go:64] FLAG: --cluster-dns="[]" Oct 29 22:42:26.100426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094116 4821 flags.go:64] FLAG: --cluster-domain="" Oct 29 22:42:26.100426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094119 4821 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 29 22:42:26.100426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094122 4821 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 29 22:42:26.101025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094126 4821 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 29 22:42:26.101025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094129 4821 flags.go:64] FLAG: --container-log-max-files="5" Oct 29 22:42:26.101025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094133 4821 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 29 22:42:26.101025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094136 4821 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 29 22:42:26.101025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094139 4821 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 29 22:42:26.101025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094142 4821 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 29 22:42:26.101025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094146 4821 flags.go:64] FLAG: --contention-profiling="false" Oct 29 22:42:26.101025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094149 4821 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 29 22:42:26.101025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094152 4821 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 29 22:42:26.101025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094155 4821 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 29 22:42:26.101025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094158 4821 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 29 22:42:26.101025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094162 4821 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 29 22:42:26.101025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094165 4821 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 29 22:42:26.101025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094168 4821 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 29 22:42:26.101025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094171 4821 flags.go:64] FLAG: --enable-load-reader="false" Oct 29 22:42:26.101025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094174 4821 flags.go:64] FLAG: --enable-server="true" Oct 29 22:42:26.101025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094177 4821 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 29 22:42:26.101025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094181 4821 flags.go:64] FLAG: --event-burst="100" Oct 29 22:42:26.101025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094185 4821 flags.go:64] FLAG: --event-qps="50" Oct 29 22:42:26.101025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094188 4821 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 29 22:42:26.101025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094192 4821 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 29 22:42:26.101025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094210 4821 flags.go:64] FLAG: --eviction-hard="" Oct 29 22:42:26.101025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094214 4821 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 29 22:42:26.101025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094218 4821 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 29 22:42:26.101645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094221 4821 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 29 22:42:26.101645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094224 4821 flags.go:64] FLAG: --eviction-soft="" Oct 29 22:42:26.101645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094227 4821 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 29 22:42:26.101645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094230 4821 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 29 22:42:26.101645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094233 4821 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 29 22:42:26.101645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094236 4821 flags.go:64] FLAG: --experimental-mounter-path="" Oct 29 22:42:26.101645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094239 4821 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 29 22:42:26.101645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094242 4821 flags.go:64] FLAG: --fail-swap-on="true" Oct 29 22:42:26.101645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094245 4821 flags.go:64] FLAG: --feature-gates="" Oct 29 22:42:26.101645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094249 4821 flags.go:64] FLAG: --file-check-frequency="20s" Oct 29 22:42:26.101645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094252 4821 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 29 22:42:26.101645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094255 4821 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 29 22:42:26.101645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094258 4821 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 29 22:42:26.101645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094261 4821 flags.go:64] FLAG: --healthz-port="10248" Oct 29 22:42:26.101645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094264 4821 flags.go:64] FLAG: --help="false" Oct 29 22:42:26.101645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094267 4821 flags.go:64] FLAG: --hostname-override="ip-10-0-130-252.ec2.internal" Oct 29 22:42:26.101645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094270 4821 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 29 22:42:26.101645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094273 4821 flags.go:64] FLAG: --http-check-frequency="20s" Oct 29 22:42:26.101645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094276 4821 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 29 22:42:26.101645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094280 4821 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 29 22:42:26.101645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094283 4821 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 29 22:42:26.101645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094286 4821 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 29 22:42:26.101645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094289 4821 flags.go:64] FLAG: --image-service-endpoint="" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094291 4821 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094294 4821 flags.go:64] FLAG: --kube-api-burst="100" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094297 4821 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094301 4821 flags.go:64] FLAG: --kube-api-qps="50" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094303 4821 flags.go:64] FLAG: --kube-reserved="" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094311 4821 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094315 4821 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094318 4821 flags.go:64] FLAG: --kubelet-cgroups="" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094321 4821 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094325 4821 flags.go:64] FLAG: --lock-file="" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094327 4821 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094331 4821 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094334 4821 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094340 4821 flags.go:64] FLAG: --log-json-split-stream="false" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094343 4821 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094347 4821 flags.go:64] FLAG: --log-text-split-stream="false" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094350 4821 flags.go:64] FLAG: --logging-format="text" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094353 4821 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094356 4821 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094359 4821 flags.go:64] FLAG: --manifest-url="" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094362 4821 flags.go:64] FLAG: --manifest-url-header="" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094367 4821 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094370 4821 flags.go:64] FLAG: --max-open-files="1000000" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094374 4821 flags.go:64] FLAG: --max-pods="110" Oct 29 22:42:26.102210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094377 4821 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094380 4821 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094383 4821 flags.go:64] FLAG: --memory-manager-policy="None" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094386 4821 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094389 4821 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094392 4821 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094395 4821 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094402 4821 flags.go:64] FLAG: --node-status-max-images="50" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094405 4821 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094408 4821 flags.go:64] FLAG: --oom-score-adj="-999" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094411 4821 flags.go:64] FLAG: --pod-cidr="" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094414 4821 flags.go:64] FLAG: --pod-manifest-path="" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094417 4821 flags.go:64] FLAG: --pod-max-pids="-1" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094421 4821 flags.go:64] FLAG: --pods-per-core="0" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094425 4821 flags.go:64] FLAG: --port="10250" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094428 4821 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094431 4821 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04f821dbc8a676187" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094434 4821 flags.go:64] FLAG: --qos-reserved="" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094437 4821 flags.go:64] FLAG: --read-only-port="10255" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094440 4821 flags.go:64] FLAG: --register-node="true" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094443 4821 flags.go:64] FLAG: --register-with-taints="" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094447 4821 flags.go:64] FLAG: --registry-burst="10" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094450 4821 flags.go:64] FLAG: --registry-qps="5" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094452 4821 flags.go:64] FLAG: --reserved-cpus="" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094456 4821 flags.go:64] FLAG: --reserved-memory="" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094460 4821 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 29 22:42:26.102827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094463 4821 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094466 4821 flags.go:64] FLAG: --rotate-certificates="false" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094469 4821 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094472 4821 flags.go:64] FLAG: --runonce="false" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094475 4821 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094478 4821 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094481 4821 flags.go:64] FLAG: --seccomp-default="false" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094484 4821 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094487 4821 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094490 4821 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094493 4821 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094496 4821 flags.go:64] FLAG: --storage-driver-password="root" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094499 4821 flags.go:64] FLAG: --storage-driver-secure="false" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094502 4821 flags.go:64] FLAG: --storage-driver-table="stats" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094505 4821 flags.go:64] FLAG: --storage-driver-user="root" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094507 4821 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094511 4821 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094513 4821 flags.go:64] FLAG: --system-cgroups="" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094516 4821 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094523 4821 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094526 4821 flags.go:64] FLAG: --tls-cert-file="" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094529 4821 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094533 4821 flags.go:64] FLAG: --tls-min-version="" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094536 4821 flags.go:64] FLAG: --tls-private-key-file="" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094538 4821 flags.go:64] FLAG: --topology-manager-policy="none" Oct 29 22:42:26.103488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094542 4821 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 29 22:42:26.104090 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094545 4821 flags.go:64] FLAG: --topology-manager-scope="container" Oct 29 22:42:26.104090 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094548 4821 flags.go:64] FLAG: --v="2" Oct 29 22:42:26.104090 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094552 4821 flags.go:64] FLAG: --version="false" Oct 29 22:42:26.104090 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094556 4821 flags.go:64] FLAG: --vmodule="" Oct 29 22:42:26.104090 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094560 4821 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 29 22:42:26.104090 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.094564 4821 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 29 22:42:26.104090 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094688 4821 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:42:26.104090 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094692 4821 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:42:26.104090 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094695 4821 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:42:26.104090 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094698 4821 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:42:26.104090 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094700 4821 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:42:26.104090 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094703 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:42:26.104090 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094705 4821 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:42:26.104090 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094708 4821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:42:26.104090 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094710 4821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:42:26.104090 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094713 4821 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:42:26.104090 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094715 4821 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:42:26.104090 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094717 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:42:26.104090 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094720 4821 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:42:26.104090 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094723 4821 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:42:26.104090 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094725 4821 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:42:26.104625 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094728 4821 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:42:26.104625 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094730 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:42:26.104625 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094733 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:42:26.104625 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094735 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:42:26.104625 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094739 4821 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:42:26.104625 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094741 4821 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:42:26.104625 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094744 4821 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:42:26.104625 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094747 4821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:42:26.104625 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094749 4821 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:42:26.104625 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094752 4821 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:42:26.104625 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094755 4821 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:42:26.104625 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094757 4821 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:42:26.104625 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094760 4821 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:42:26.104625 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094762 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:42:26.104625 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094781 4821 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:42:26.104625 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094785 4821 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:42:26.104625 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094789 4821 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:42:26.104625 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094793 4821 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:42:26.104625 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094796 4821 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:42:26.104625 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094799 4821 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:42:26.105116 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094802 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:42:26.105116 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094804 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:42:26.105116 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094807 4821 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:42:26.105116 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094809 4821 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:42:26.105116 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094811 4821 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:42:26.105116 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094814 4821 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:42:26.105116 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094816 4821 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:42:26.105116 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094819 4821 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:42:26.105116 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094821 4821 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:42:26.105116 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094824 4821 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:42:26.105116 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094826 4821 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:42:26.105116 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094829 4821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:42:26.105116 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094832 4821 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:42:26.105116 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094835 4821 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:42:26.105116 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094838 4821 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:42:26.105116 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094841 4821 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:42:26.105116 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094844 4821 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:42:26.105116 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094847 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:42:26.105116 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094849 4821 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:42:26.105116 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094852 4821 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:42:26.105632 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094856 4821 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:42:26.105632 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094858 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:42:26.105632 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094861 4821 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:42:26.105632 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094863 4821 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:42:26.105632 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094866 4821 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:42:26.105632 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094868 4821 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:42:26.105632 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094873 4821 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:42:26.105632 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094876 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:42:26.105632 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094879 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:42:26.105632 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094882 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:42:26.105632 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094885 4821 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:42:26.105632 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094888 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:42:26.105632 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094891 4821 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:42:26.105632 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094894 4821 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:42:26.105632 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094896 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:42:26.105632 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094899 4821 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:42:26.105632 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094901 4821 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:42:26.105632 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094904 4821 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:42:26.106147 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094906 4821 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:42:26.106147 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094909 4821 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:42:26.106147 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094911 4821 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:42:26.106147 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094914 4821 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:42:26.106147 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094916 4821 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:42:26.106147 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094919 4821 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:42:26.106147 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094922 4821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:42:26.106147 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094924 4821 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:42:26.106147 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094926 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:42:26.106147 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094929 4821 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:42:26.106147 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094932 4821 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:42:26.106147 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094935 4821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:42:26.106147 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094937 4821 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:42:26.106147 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094940 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:42:26.106147 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094944 4821 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:42:26.106147 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094946 4821 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:42:26.106147 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094949 4821 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:42:26.106147 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094951 4821 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:42:26.106147 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094954 4821 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:42:26.106147 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094956 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:42:26.106714 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094959 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:42:26.106714 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094961 4821 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:42:26.106714 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094964 4821 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:42:26.106714 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094968 4821 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:42:26.106714 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094971 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:42:26.106714 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094974 4821 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:42:26.106714 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094977 4821 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:42:26.106714 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094980 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:42:26.106714 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094982 4821 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:42:26.106714 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094985 4821 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:42:26.106714 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094987 4821 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:42:26.106714 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094990 4821 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:42:26.106714 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.094993 4821 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:42:26.106714 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.095017 4821 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:42:26.106714 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095057 4821 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:42:26.107099 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095061 4821 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:42:26.107099 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095064 4821 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:42:26.107099 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095067 4821 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:42:26.107099 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095069 4821 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:42:26.107099 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095072 4821 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:42:26.107099 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095075 4821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:42:26.107099 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095077 4821 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:42:26.107099 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095081 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:42:26.107099 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095084 4821 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:42:26.107099 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095086 4821 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:42:26.107099 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095089 4821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:42:26.107099 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095091 4821 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:42:26.107099 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095095 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:42:26.107099 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095098 4821 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:42:26.107099 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095100 4821 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:42:26.107099 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095103 4821 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:42:26.107099 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095105 4821 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:42:26.107099 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095109 4821 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:42:26.107099 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095111 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:42:26.107099 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095114 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:42:26.107627 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095116 4821 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:42:26.107627 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095119 4821 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:42:26.107627 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095123 4821 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:42:26.107627 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095127 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:42:26.107627 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095130 4821 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:42:26.107627 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095133 4821 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:42:26.107627 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095135 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:42:26.107627 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095138 4821 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:42:26.107627 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095140 4821 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:42:26.107627 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095143 4821 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:42:26.107627 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095146 4821 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:42:26.107627 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095148 4821 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:42:26.107627 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095150 4821 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:42:26.107627 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095153 4821 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:42:26.107627 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095156 4821 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:42:26.107627 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095158 4821 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:42:26.107627 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095161 4821 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:42:26.107627 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095163 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:42:26.107627 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095165 4821 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:42:26.108154 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095169 4821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:42:26.108154 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095172 4821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:42:26.108154 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095174 4821 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:42:26.108154 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095177 4821 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:42:26.108154 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095179 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:42:26.108154 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095185 4821 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:42:26.108154 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095187 4821 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:42:26.108154 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095190 4821 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:42:26.108154 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095193 4821 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:42:26.108154 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095208 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:42:26.108154 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095211 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:42:26.108154 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095214 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:42:26.108154 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095216 4821 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:42:26.108154 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095219 4821 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:42:26.108154 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095221 4821 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:42:26.108154 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095224 4821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:42:26.108154 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095227 4821 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:42:26.108154 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095230 4821 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:42:26.108154 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095232 4821 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:42:26.108154 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095235 4821 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:42:26.108731 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095238 4821 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:42:26.108731 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095240 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:42:26.108731 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095243 4821 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:42:26.108731 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095245 4821 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:42:26.108731 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095248 4821 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:42:26.108731 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095250 4821 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:42:26.108731 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095254 4821 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:42:26.108731 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095256 4821 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:42:26.108731 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095259 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:42:26.108731 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095261 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:42:26.108731 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095264 4821 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:42:26.108731 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095266 4821 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:42:26.108731 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095270 4821 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:42:26.108731 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095272 4821 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:42:26.108731 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095275 4821 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:42:26.108731 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095277 4821 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:42:26.108731 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095280 4821 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:42:26.108731 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095283 4821 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:42:26.108731 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095286 4821 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:42:26.109268 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095289 4821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:42:26.109268 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095291 4821 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:42:26.109268 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095294 4821 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:42:26.109268 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095296 4821 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:42:26.109268 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095299 4821 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:42:26.109268 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095302 4821 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:42:26.109268 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095304 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:42:26.109268 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095307 4821 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:42:26.109268 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095310 4821 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:42:26.109268 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095312 4821 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:42:26.109268 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095315 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:42:26.109268 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095318 4821 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:42:26.109268 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095321 4821 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:42:26.109268 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095323 4821 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:42:26.109268 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095326 4821 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:42:26.109268 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095330 4821 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:42:26.109268 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095333 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:42:26.109268 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095337 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:42:26.109268 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095340 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:42:26.109825 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095343 4821 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:42:26.109825 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095345 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:42:26.109825 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095348 4821 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:42:26.109825 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095352 4821 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:42:26.109825 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095354 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:42:26.109825 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095357 4821 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:42:26.109825 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095360 4821 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:42:26.109825 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.095362 4821 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:42:26.109825 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.100672 4821 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 29 22:42:26.109825 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.100689 4821 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 29 22:42:26.109825 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.100707 4821 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 29 22:42:26.109825 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.100713 4821 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 29 22:42:26.109825 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100773 4821 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:42:26.109825 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100778 4821 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:42:26.109825 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100781 4821 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:42:26.109825 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100784 4821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:42:26.109825 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100787 4821 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:42:26.109825 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100791 4821 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:42:26.109825 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100795 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:42:26.110308 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100799 4821 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:42:26.110308 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100801 4821 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:42:26.110308 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100804 4821 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:42:26.110308 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100807 4821 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:42:26.110308 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100809 4821 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:42:26.110308 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100812 4821 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:42:26.110308 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100815 4821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:42:26.110308 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100818 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:42:26.110308 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100821 4821 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:42:26.110308 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100823 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:42:26.110308 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100826 4821 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:42:26.110308 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100829 4821 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:42:26.110308 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100831 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:42:26.110308 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100834 4821 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:42:26.110308 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100836 4821 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:42:26.110308 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100841 4821 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:42:26.110308 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100845 4821 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:42:26.110308 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100848 4821 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:42:26.110308 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100851 4821 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:42:26.110777 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100855 4821 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:42:26.110777 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100857 4821 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:42:26.110777 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100861 4821 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:42:26.110777 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100864 4821 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:42:26.110777 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100867 4821 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:42:26.110777 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100869 4821 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:42:26.110777 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100872 4821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:42:26.110777 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100875 4821 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:42:26.110777 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100877 4821 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:42:26.110777 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100880 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:42:26.110777 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100882 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:42:26.110777 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100885 4821 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:42:26.110777 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100887 4821 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:42:26.110777 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100890 4821 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:42:26.110777 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100892 4821 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:42:26.110777 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100894 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:42:26.110777 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100897 4821 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:42:26.110777 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100899 4821 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:42:26.110777 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100902 4821 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:42:26.110777 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100904 4821 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:42:26.111295 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100907 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:42:26.111295 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100910 4821 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:42:26.111295 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100912 4821 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:42:26.111295 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100915 4821 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:42:26.111295 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100918 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:42:26.111295 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100921 4821 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:42:26.111295 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100923 4821 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:42:26.111295 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100926 4821 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:42:26.111295 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100928 4821 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:42:26.111295 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100931 4821 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:42:26.111295 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100934 4821 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:42:26.111295 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100937 4821 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:42:26.111295 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100939 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:42:26.111295 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100942 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:42:26.111295 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100945 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:42:26.111295 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100947 4821 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:42:26.111295 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100950 4821 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:42:26.111295 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100952 4821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:42:26.111295 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100955 4821 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:42:26.111295 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100957 4821 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:42:26.111782 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100960 4821 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:42:26.111782 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100962 4821 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:42:26.111782 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100965 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:42:26.111782 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100968 4821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:42:26.111782 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100971 4821 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:42:26.111782 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100973 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:42:26.111782 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100975 4821 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:42:26.111782 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100978 4821 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:42:26.111782 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100981 4821 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:42:26.111782 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100983 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:42:26.111782 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100986 4821 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:42:26.111782 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100988 4821 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:42:26.111782 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100991 4821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:42:26.111782 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100994 4821 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:42:26.111782 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.100997 4821 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:42:26.111782 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101000 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:42:26.111782 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101002 4821 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:42:26.111782 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101005 4821 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:42:26.111782 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101007 4821 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:42:26.111782 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101010 4821 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:42:26.112412 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101013 4821 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:42:26.112412 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101015 4821 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:42:26.112412 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101018 4821 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:42:26.112412 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101020 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:42:26.112412 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101023 4821 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:42:26.112412 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101025 4821 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:42:26.112412 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101028 4821 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:42:26.112412 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101031 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:42:26.112412 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101034 4821 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:42:26.112412 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101036 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:42:26.112412 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101039 4821 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:42:26.112412 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101042 4821 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:42:26.112412 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101045 4821 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:42:26.112412 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101047 4821 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:42:26.112412 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101050 4821 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:42:26.112412 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101052 4821 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:42:26.112412 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101055 4821 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:42:26.112412 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101057 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:42:26.112412 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101059 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:42:26.112412 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101062 4821 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:42:26.112926 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.101088 4821 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:42:26.112926 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101253 4821 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:42:26.112926 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101259 4821 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:42:26.112926 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101263 4821 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:42:26.112926 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101266 4821 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:42:26.112926 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101270 4821 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:42:26.112926 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101273 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:42:26.112926 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101276 4821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:42:26.112926 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101279 4821 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:42:26.112926 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101282 4821 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:42:26.112926 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101284 4821 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:42:26.112926 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101287 4821 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:42:26.112926 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101290 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:42:26.112926 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101292 4821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:42:26.112926 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101295 4821 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:42:26.112926 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101298 4821 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:42:26.113340 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101300 4821 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:42:26.113340 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101303 4821 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:42:26.113340 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101305 4821 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:42:26.113340 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101308 4821 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:42:26.113340 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101310 4821 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:42:26.113340 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101313 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:42:26.113340 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101316 4821 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:42:26.113340 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101318 4821 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:42:26.113340 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101321 4821 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:42:26.113340 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101323 4821 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:42:26.113340 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101326 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:42:26.113340 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101328 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:42:26.113340 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101331 4821 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:42:26.113340 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101334 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:42:26.113340 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101336 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:42:26.113340 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101339 4821 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:42:26.113340 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101341 4821 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:42:26.113340 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101344 4821 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:42:26.113340 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101347 4821 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:42:26.113340 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101349 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:42:26.113820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101352 4821 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:42:26.113820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101355 4821 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:42:26.113820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101357 4821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:42:26.113820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101360 4821 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:42:26.113820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101363 4821 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:42:26.113820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101366 4821 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:42:26.113820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101368 4821 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:42:26.113820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101371 4821 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:42:26.113820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101373 4821 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:42:26.113820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101376 4821 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:42:26.113820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101379 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:42:26.113820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101382 4821 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:42:26.113820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101384 4821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:42:26.113820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101387 4821 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:42:26.113820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101389 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:42:26.113820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101392 4821 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:42:26.113820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101394 4821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:42:26.113820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101396 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:42:26.113820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101399 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:42:26.113820 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101402 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:42:26.114520 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101404 4821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:42:26.114520 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101406 4821 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:42:26.114520 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101409 4821 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:42:26.114520 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101411 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:42:26.114520 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101414 4821 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:42:26.114520 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101416 4821 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:42:26.114520 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101419 4821 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:42:26.114520 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101422 4821 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:42:26.114520 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101424 4821 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:42:26.114520 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101427 4821 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:42:26.114520 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101430 4821 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:42:26.114520 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101432 4821 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:42:26.114520 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101435 4821 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:42:26.114520 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101437 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:42:26.114520 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101441 4821 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:42:26.114520 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101445 4821 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:42:26.114520 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101449 4821 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:42:26.114520 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101452 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:42:26.114520 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101455 4821 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:42:26.115269 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101458 4821 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:42:26.115269 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101461 4821 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:42:26.115269 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101463 4821 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:42:26.115269 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101466 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:42:26.115269 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101469 4821 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:42:26.115269 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101471 4821 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:42:26.115269 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101474 4821 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:42:26.115269 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101476 4821 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:42:26.115269 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101479 4821 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:42:26.115269 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101481 4821 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:42:26.115269 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101484 4821 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:42:26.115269 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101486 4821 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:42:26.115269 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101489 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:42:26.115269 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101491 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:42:26.115269 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101494 4821 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:42:26.115269 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101496 4821 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:42:26.115269 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101499 4821 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:42:26.115269 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101501 4821 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:42:26.115269 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101504 4821 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:42:26.115269 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101506 4821 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:42:26.115767 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101509 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:42:26.115767 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101511 4821 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:42:26.115767 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101514 4821 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:42:26.115767 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101516 4821 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:42:26.115767 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101519 4821 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:42:26.115767 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101521 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:42:26.115767 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101524 4821 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:42:26.115767 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101527 4821 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:42:26.115767 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101531 4821 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:42:26.115767 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101535 4821 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:42:26.115767 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101538 4821 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:42:26.115767 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101541 4821 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:42:26.115767 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.101564 4821 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:42:26.115767 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101596 4821 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:42:26.115767 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101600 4821 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:42:26.116137 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101603 4821 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:42:26.116137 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101605 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:42:26.116137 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101608 4821 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:42:26.116137 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101611 4821 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:42:26.116137 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101613 4821 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:42:26.116137 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101616 4821 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:42:26.116137 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101618 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:42:26.116137 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101621 4821 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:42:26.116137 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101624 4821 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:42:26.116137 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101627 4821 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:42:26.116137 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101633 4821 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:42:26.116137 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101635 4821 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:42:26.116137 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101638 4821 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:42:26.116137 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101641 4821 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:42:26.116137 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101643 4821 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:42:26.116137 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101646 4821 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:42:26.116137 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101649 4821 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:42:26.116137 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101652 4821 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:42:26.116137 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101654 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:42:26.116715 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101657 4821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:42:26.116715 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101660 4821 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:42:26.116715 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101662 4821 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:42:26.116715 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101665 4821 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:42:26.116715 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101667 4821 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:42:26.116715 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101670 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:42:26.116715 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101673 4821 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:42:26.116715 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101675 4821 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:42:26.116715 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101678 4821 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:42:26.116715 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101682 4821 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:42:26.116715 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101684 4821 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:42:26.116715 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101687 4821 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:42:26.116715 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101690 4821 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:42:26.116715 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101692 4821 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:42:26.116715 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101695 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:42:26.116715 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101697 4821 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:42:26.116715 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101700 4821 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:42:26.116715 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101702 4821 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:42:26.116715 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101705 4821 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:42:26.116715 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101707 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:42:26.117237 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101710 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:42:26.117237 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101712 4821 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:42:26.117237 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101715 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:42:26.117237 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101718 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:42:26.117237 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101721 4821 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:42:26.117237 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101724 4821 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:42:26.117237 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101726 4821 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:42:26.117237 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101729 4821 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:42:26.117237 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101732 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:42:26.117237 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101734 4821 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:42:26.117237 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101737 4821 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:42:26.117237 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101740 4821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:42:26.117237 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101742 4821 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:42:26.117237 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101744 4821 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:42:26.117237 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101747 4821 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:42:26.117237 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101750 4821 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:42:26.117237 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101752 4821 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:42:26.117237 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101755 4821 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:42:26.117237 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101757 4821 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:42:26.117711 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101760 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:42:26.117711 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101763 4821 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:42:26.117711 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101765 4821 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:42:26.117711 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101768 4821 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:42:26.117711 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101771 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:42:26.117711 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101773 4821 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:42:26.117711 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101775 4821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:42:26.117711 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101778 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:42:26.117711 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101780 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:42:26.117711 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101783 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:42:26.117711 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101785 4821 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:42:26.117711 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101788 4821 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:42:26.117711 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101790 4821 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:42:26.117711 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101793 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:42:26.117711 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101795 4821 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:42:26.117711 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101798 4821 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:42:26.117711 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101801 4821 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:42:26.117711 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101804 4821 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:42:26.117711 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101806 4821 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:42:26.118158 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101808 4821 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:42:26.118158 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101811 4821 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:42:26.118158 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101814 4821 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:42:26.118158 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101816 4821 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:42:26.118158 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101818 4821 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:42:26.118158 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101822 4821 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:42:26.118158 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101826 4821 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:42:26.118158 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101829 4821 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:42:26.118158 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101832 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:42:26.118158 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101834 4821 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:42:26.118158 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101837 4821 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:42:26.118158 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101839 4821 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:42:26.118158 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101842 4821 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:42:26.118158 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101845 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:42:26.118158 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101847 4821 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:42:26.118158 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101850 4821 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:42:26.118158 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101853 4821 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:42:26.118158 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101855 4821 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:42:26.118158 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101858 4821 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:42:26.118633 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101860 4821 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:42:26.118633 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101864 4821 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:42:26.118633 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101866 4821 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:42:26.118633 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101869 4821 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:42:26.118633 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101871 4821 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:42:26.118633 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101874 4821 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:42:26.118633 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101876 4821 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:42:26.118633 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:42:26.101878 4821 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:42:26.118633 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.101929 4821 server.go:957] "Client rotation is on, will bootstrap in background" Oct 29 22:42:26.118633 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.102934 4821 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 29 22:42:26.118633 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.102992 4821 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 29 22:42:26.118633 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.103166 4821 server.go:1014] "Starting client certificate rotation" Oct 29 22:42:26.118633 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.103304 4821 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 22:42:26.118633 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.103362 4821 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:36:46 +0000 UTC" deadline="2028-07-30 23:54:03.446589978 +0000 UTC" Oct 29 22:42:26.118633 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.103405 4821 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15361h11m37.343188471s" Oct 29 22:42:26.118633 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.103737 4821 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:42:26.119170 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.104853 4821 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:42:26.119170 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.106245 4821 log.go:25] "Validated CRI v1 runtime API" Oct 29 22:42:26.119170 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.107888 4821 log.go:25] "Validated CRI v1 image API" Oct 29 22:42:26.119170 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.108564 4821 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 29 22:42:26.119170 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.110120 4821 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7aac7e64-9e19-4146-b16a-f90e6d54c167:/dev/nvme0n1p3 f8d99165-074d-4f21-86dd-53b3f464fb7c:/dev/nvme0n1p4] Oct 29 22:42:26.119344 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.110139 4821 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1135b6d0397687f9487e54cfc133c8d6a93b6b9cccf91599c4b75ef7c3cddf85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1135b6d0397687f9487e54cfc133c8d6a93b6b9cccf91599c4b75ef7c3cddf85/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16891bc02d310862d1fc552da31b7cc0205b945c4d2f2479f1b95eafdbaed289/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16891bc02d310862d1fc552da31b7cc0205b945c4d2f2479f1b95eafdbaed289/userdata/shm major:0 minor:272 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16b261e674b5fc0221acefaf8aa0ed6b1b98af78187d75b09e1006787f60a040/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16b261e674b5fc0221acefaf8aa0ed6b1b98af78187d75b09e1006787f60a040/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/256d11bdaba8884eeb5c071b63c56ee8fbe9b4f8b79ca262d8b6727b9d4e3353/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/256d11bdaba8884eeb5c071b63c56ee8fbe9b4f8b79ca262d8b6727b9d4e3353/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/43eb1690a2264835c978f4dcc58bf4cfc8c17d25c0ff29a4f73e3f8922099f95/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/43eb1690a2264835c978f4dcc58bf4cfc8c17d25c0ff29a4f73e3f8922099f95/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5738903c517750fdd3189dd5853f08e305912b8e74f0504ec99dde0ffbb5d401/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5738903c517750fdd3189dd5853f08e305912b8e74f0504ec99dde0ffbb5d401/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59ab3ed9c9fb828acdc99da907d98147baa0b8793345fd3d6c9f8e932ef21779/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59ab3ed9c9fb828acdc99da907d98147baa0b8793345fd3d6c9f8e932ef21779/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5fe6c4845b6d870cedbb00207a211c8cb9216002c9047e6175f2e691cac77de4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5fe6c4845b6d870cedbb00207a211c8cb9216002c9047e6175f2e691cac77de4/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8258a9c17fdfce96ec8fdaa0b7d8b53e69dd050c3f578c21279eca26a2ded146/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8258a9c17fdfce96ec8fdaa0b7d8b53e69dd050c3f578c21279eca26a2ded146/userdata/shm major:0 minor:99 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9597c7254279a4acc36b251def7dec3cb05d5ecffc4b49fe9fc95a4bcdeb70c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9597c7254279a4acc36b251def7dec3cb05d5ecffc4b49fe9fc95a4bcdeb70c7/userdata/shm major:0 minor:274 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9df89bff6b79e8cfaddaf04603087a596c31dbbb8591f9c9d0abb9e16da76f5d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9df89bff6b79e8cfaddaf04603087a596c31dbbb8591f9c9d0abb9e16da76f5d/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a32d691bc1c503e421d0c7cade812b5ed084aa8b5ffcda1cfa550ec243d2c1e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a32d691bc1c503e421d0c7cade812b5ed084aa8b5ffcda1cfa550ec243d2c1e4/userdata/shm major:0 minor:106 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa2507ccc88a01b0ef9b0e5bb4d7b8c4ab759816bfdac374fac1e12a2437e2a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa2507ccc88a01b0ef9b0e5bb4d7b8c4ab759816bfdac374fac1e12a2437e2a2/userdata/shm major:0 minor:100 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/addbe92aa928a46f8b9299d8e767868b40ac40aa9b24142658573912c660c257/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/addbe92aa928a46f8b9299d8e767868b40ac40aa9b24142658573912c660c257/userdata/shm major:0 minor:269 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/be01642a04bd5f86f0f4682b268dea238129b34cadf8482f8ab0c03cdf7f27b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/be01642a04bd5f86f0f4682b268dea238129b34cadf8482f8ab0c03cdf7f27b5/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef45efa92af3dab51ceeb174fdbcec84aa5201e24deda9837a403933bd21f0a0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef45efa92af3dab51ceeb174fdbcec84aa5201e24deda9837a403933bd21f0a0/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f814ce24c2e1df862efd858815d8f17a65493d2b023fa5bdce3352d728002867/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f814ce24c2e1df862efd858815d8f17a65493d2b023fa5bdce3352d728002867/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0061f2cf-c501-47b0-a89b-3e244f9c313d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/0061f2cf-c501-47b0-a89b-3e244f9c313d/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ae4b768-8a6b-4c10-ae02-c1a3950d830c/volumes/kubernetes.io~projected/kube-api-access-hhdc6:{mountpoint:/var/lib/kubelet/pods/1ae4b768-8a6b-4c10-ae02-c1a3950d830c/volumes/kubernetes.io~projected/kube-api-access-hhdc6 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ae4b768-8a6b-4c10-ae02-c1a3950d830c/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/1ae4b768-8a6b-4c10-ae02-c1a3950d830c/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33360cd4-4738-4c3d-bc9f-9f5003408589/volumes/kubernetes.io~projected/kube-api-access-fmn55:{mountpoint:/var/lib/kubelet/pods/33360cd4-4738-4c3d-bc9f-9f5003408589/volumes/kubernetes.io~projected/kube-api-access-fmn55 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4026e493-67d9-4b2c-8267-1da3a0b1d621/volumes/kubernetes.io~projected/kube-api-access-hplpk:{mountpoint:/var/lib/kubelet/pods/4026e493-67d9-4b2c-8267-1da3a0b1d621/volumes/kubernetes.io~projected/kube-api-access-hplpk major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4026e493-67d9-4b2c-8267-1da3a0b1d621/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/4026e493-67d9-4b2c-8267-1da3a0b1d621/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4916614f-8a34-491d-b4ad-bae72c14b140/volumes/kubernetes.io~projected/kube-api-access-6d5dd:{mountpoint:/var/lib/kubelet/pods/4916614f-8a34-491d-b4ad-bae72c14b140/volumes/kubernetes.io~projected/kube-api-access-6d5dd major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4916614f-8a34-491d-b4ad-bae72c14b140/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/4916614f-8a34-491d-b4ad-bae72c14b140/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4916614f-8a34-491d-b4ad-bae72c14b140/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/4916614f-8a34-491d-b4ad-bae72c14b140/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51fcfffe-f89d-4064-a0e3-45461cc62570/volumes/kubernetes.io~projected/kube-api-access-wd8z9:{mountpoint:/var/lib/kubelet/pods/51fcfffe-f89d-4064-a0e3-45461cc62570/volumes/kubernetes.io~projected/kube-api-access-wd8z9 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ea8559a-aa64-4d00-84c0-2c9405e32606/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/6ea8559a-aa64-4d00-84c0-2c9405e32606/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ea8559a-aa64-4d00-84c0-2c9405e32606/volumes/kubernetes.io~projected/kube-api-access-klzkm:{mountpoint:/var/lib/kubelet/pods/6ea8559a-aa64-4d00-84c0-2c9405e32606/volumes/kubernetes.io~projected/kube-api-access-klzkm major:0 minor:70 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ea8559a-aa64-4d00-84c0-2c9405e32606/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/6ea8559a-aa64-4d00-84c0-2c9405e32606/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99a7b227-aad8-414f-8668-70a5660d6a6e/volumes/kubernetes.io~projected/kube-api-access-qvp94:{mountpoint:/var/lib/kubelet/pods/99a7b227-aad8-414f-8668-70a5660d6a6e/volumes/kubernetes.io~projected/kube-api-access-qvp94 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8ad4438-da5a-4f3a-bb5b-07c330aef792/volumes/kubernetes.io~projected/kube-api-access-x4845:{mountpoint:/var/lib/kubelet/pods/a8ad4438-da5a-4f3a-bb5b-07c330aef792/volumes/kubernetes.io~projected/kube-api-access-x4845 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8ad4438-da5a-4f3a-bb5b-07c330aef792/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/a8ad4438-da5a-4f3a-bb5b-07c330aef792/volumes/kubernetes.io~secret/metrics-certs major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af139c4a-79a2-47b4-95e4-493e3430be98/volumes/kubernetes.io~projected/kube-api-access-9w9d9:{mountpoint:/var/lib/kubelet/pods/af139c4a-79a2-47b4-95e4-493e3430be98/volumes/kubernetes.io~projected/kube-api-access-9w9d9 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff/volumes/kubernetes.io~projected/kube-api-access-f6ll2:{mountpoint:/var/lib/kubelet/pods/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff/volumes/kubernetes.io~projected/kube-api-access-f6ll2 major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c147a7be-d6f6-458e-8b0d-70da7a3c7026/volumes/kubernetes.io~projected/kube-api-access-qb8pj:{mountpoint:/var/lib/kubelet/pods/c147a7be-d6f6-458e-8b0d-70da7a3c7026/volumes/kubernetes.io~projected/kube-api-access-qb8pj major:0 minor:68 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dab4a141-b01a-4434-a792-d8fc9d63a83d/volumes/kubernetes.io~projected/kube-api-access-k7tgw:{mountpoint:/var/lib/kubelet/pods/dab4a141-b01a-4434-a792-d8fc9d63a83d/volumes/kubernetes.io~projected/kube-api-access-k7tgw major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dab4a141-b01a-4434-a792-d8fc9d63a83d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/dab4a141-b01a-4434-a792-d8fc9d63a83d/volumes/kubernetes.io~secret/cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1/volumes/kubernetes.io~projected/kube-api-access-45m2j:{mountpoint:/var/lib/kubelet/pods/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1/volumes/kubernetes.io~projected/kube-api-access-45m2j major:0 minor:69 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f99954e5-6dfe-457f-9a8f-22752798bb5d/volumes/kubernetes.io~projected/kube-api-access-hsddk:{mountpoint:/var/lib/kubelet/pods/f99954e5-6dfe-457f-9a8f-22752798bb5d/volumes/kubernetes.io~projected/kube-api-access-hsddk major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd35650d-2c53-4230-8518-c4cb05b59558/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/fd35650d-2c53-4230-8518-c4cb05b59558/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:268 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/ffb1e31faf23fe165838f7ead0049329d8a075ac1c947cfa9d79d7b43ddd908e/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/616dd26c514a0260cbdebcdebd8c2b5a90e61ac8d3846df1c68258ea462d52c1/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/f7a2f8002eaa68ad4cbff2b471ed73a595f1743ac9439ea9bb54d4d2b6a05b49/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/2185408e785ad34eed07490157be707ee76e99951d0e15cf34b8690595ec6df0/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-113:{mountpoint:/var/lib/containers/storage/overlay/394a25c26b4fa6b8d30f5eb1398ed4ac7e01aac6a9dd1305151a1e07d70eaa87/merged major:0 minor:113 fsType:overlay blockSize:0} overlay_0-121:{mountpoint:/var/lib/containers/storage/overlay/d826da05764d50271a8747f24da659c0b6dff5e5cf175e47ca4a470c89b6870e/merged major:0 minor:121 fsType:overlay blockSize:0} overlay_0-123:{mountpoint:/var/lib/containers/storage/overlay/69b68ef3756176cf4c00d18f544159487d6d54aefaa03903359a5533b1a9b4e9/merged major:0 minor:123 fsType:overlay blockSize:0} overlay_0-130:{mountpoint:/var/lib/containers/storage/overlay/9073858177cce62f9b0bab8491f227db77bad0add3efe2e014649cf6dbc0e888/merged major:0 minor:130 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/b80a6aa030a0ed3f5f39a07c3ec96c158b2ece6fa2a25dd6030b005686369618/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/f5986c3cb9a5e7a5e9757b37b074ff6aac41c6c7f00868ba3dd3f4d3181afb46/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/cd7e0f99dd9924db95a5aaf864fadd80bcbe0466414a355e7f734d6b25b7edcc/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/d3e8cac85d81e144fc6fc0612b365d396240db387ab81cff09e34b29eb793634/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/c425a8fc04e433d12b9510f58a2591c6b9fa194d26c053ff45a8e0e87b682258/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-156:{mountpoint:/var/lib/containers/storage/overlay/40a17854aba75030f24450bd6363b1d2257059e2057f3bc358db46d7ab9ae009/merged major:0 minor:156 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/d0968e555752270bd667f49118a42aca8e3e19820437fedcb6cbb2a37c5f240d/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/8ac650ddcedee892a99ba5f4bf010d2d3b1e729e10d9ae0c7d06e63b3781ff1d/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-171:{mountpoint:/var/lib/containers/storage/overlay/8dcd123daba04d34baca848a38379ef05d274ad88893149078f260f16938dca1/merged major:0 minor:171 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/8175ca27035d6a0786a8fa3babaea7510ff9efb195a7d56f2a004cd12efe1712/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/ce68a4f690dc70c84494452ee2ed36b86d35abdf342c049263ef617710df1921/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-189:{mountpoint:/var/lib/containers/storage/overlay/610585e46b384f9f91786c86ff540531e8b06d2a67900d2ca6e9f98f8785e3f5/merged major:0 minor:189 fsType:overlay blockSize:0} overlay_0-194:{mountpoint:/var/lib/containers/storage/overlay/54ec19432a439755ecb02cfd43829416c74a17a826896d212807165d5e5c3035/merged major:0 minor:194 fsType:overlay blockSize:0} overlay_0-199:{mountpoint:/var/lib/containers/storage/overlay/56171a85a1d065b2887b7c60b407b25ed563180178edda535c1619905c10551a/merged major:0 minor:199 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/c8694d17dca61a1f11f4672041bf41ef5d1e11bef98ee9899b1797eb22f39a17/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/b25bc1b280bf6af9f28a2f3d63534c686a7f2d83b701010484f1a14d060c1049/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/bd1bc857beb3d46ce6ead13220e96ac8704aa2f1b83da399dc34d1f4005cca59/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/6fae17c76f58c3169f5a6199d26eb8670d790fa50c99305dad959cb4903eceb0/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/d6a06eb4e6b568e45ea4acdbd2e8e9ae80922a12286f455abf427b3f2145339f/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/ba7756d0b423e5fe5b6cd8470b3b46380c0c78a374645ab9aaf4a221cb799719/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/1fa3a81fd038564fab7ce3892436d32dc950f22e2637506b51573242830fea64/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/9160e8bd085511fe5e1622c822031494c78949fdb59594357fc28be32c00586b/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/7fd8e67e019dfaa2281b2a6219b1110c03cfa047373a875ae9f5e95368c098c5/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/73810c820ee9fdab064d41270bb20fc76c496524737020896f5e46adbef8093e/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/00b13dc8e5a37e64c31d929f5e9592bcebdbf12545f8fe32172c0c09470a1784/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/fc75fce1c8279048bd6859d962088183683d954df26127b2f3d35138a01961c4/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/1d64ccbfadabfc1cc6b570967296cd91afcdb737bdb1d6f526c33dea02f07d15/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/969bab9e71cc91e7ef13e7b7b8172170a6ab61d7132a4ba7aeb1aa3c1b1b21d9/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/4c9d47eeefa73089762df0708e7248fecc1da86e04311b73f53f89b3a6d797f8/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/cfab6b91a00dee9c6921d08ed0bc33d714ffd139a9614a76f563633834c7bb26/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/20e7f4b37ae4d00105962ea1ef42b9fb5a91999c3971dc08d3bbf87909193c7b/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/9074f0fd0d22c552afc7e1a3cfa9727febf9343f094a769d0fc9dae945dacdb9/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/f078a08184e963548550e5971d9d26ae67c811def994c946422479ad465ec104/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/e57be26ce69df358edf683a0b251b684cff1f0224778f2885defef57cb37c044/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/8bedbe42d188440a97a881ba87ce08d231bf0c94bed694e5b10e32d83c6f8e3b/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/288bae4bf534b55ba53b426b3f22c9817bc16cdb23cba5eaad109c3e8bb7e4a5/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/59cb53238a929d9dcad20004355ffc45d58d34e69322bee4f3f4162242dc5f96/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/33461670b373b91a66542d2791769346b112368b7caad934a0afb6eac42b6886/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-87:{mountpoint:/var/lib/containers/storage/overlay/8d68412703df68c14f3ec22d483d297c427d77f855b68f7a481b669bc592921f/merged major:0 minor:87 fsType:overlay blockSize:0} overlay_0-91:{mountpoint:/var/lib/containers/storage/overlay/99bb22047c4f85ce13587a0b834daf2bb1c260410cbe4ec42f6d7870244b14f6/merged major:0 minor:91 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/504fc9f552e5d04a6e682cd610bd82afe9c8cac8f250cb37e09151091998f02c/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/841a4b2e53010b3b99b3cf95777e88d83a6a36e29ef2e73ee83b7fc754953512/merged major:0 minor:97 fsType:overlay blockSize:0}] Oct 29 22:42:26.119542 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.117416 4821 manager.go:217] Machine: {Timestamp:2026-10-29 22:42:26.116942044 +0000 UTC m=+0.057489945 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097619 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23000d1d50c1d5a6b50935b5c25ef4 SystemUUID:ec23000d-1d50-c1d5-a6b5-0935b5c25ef4 BootID:311142fc-d8c7-4234-8304-06eb74f93906 Filesystems:[{Device:overlay_0-130 DeviceMajor:0 DeviceMinor:130 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-171 DeviceMajor:0 DeviceMinor:171 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff/volumes/kubernetes.io~projected/kube-api-access-f6ll2 DeviceMajor:0 DeviceMinor:271 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef45efa92af3dab51ceeb174fdbcec84aa5201e24deda9837a403933bd21f0a0/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa2507ccc88a01b0ef9b0e5bb4d7b8c4ab759816bfdac374fac1e12a2437e2a2/userdata/shm DeviceMajor:0 DeviceMinor:100 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-156 DeviceMajor:0 DeviceMinor:156 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/4916614f-8a34-491d-b4ad-bae72c14b140/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/af139c4a-79a2-47b4-95e4-493e3430be98/volumes/kubernetes.io~projected/kube-api-access-9w9d9 DeviceMajor:0 DeviceMinor:63 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5fe6c4845b6d870cedbb00207a211c8cb9216002c9047e6175f2e691cac77de4/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4026e493-67d9-4b2c-8267-1da3a0b1d621/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:214 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16891bc02d310862d1fc552da31b7cc0205b945c4d2f2479f1b95eafdbaed289/userdata/shm DeviceMajor:0 DeviceMinor:272 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/6ea8559a-aa64-4d00-84c0-2c9405e32606/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/dab4a141-b01a-4434-a792-d8fc9d63a83d/volumes/kubernetes.io~projected/kube-api-access-k7tgw DeviceMajor:0 DeviceMinor:220 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f814ce24c2e1df862efd858815d8f17a65493d2b023fa5bdce3352d728002867/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/addbe92aa928a46f8b9299d8e767868b40ac40aa9b24142658573912c660c257/userdata/shm DeviceMajor:0 DeviceMinor:269 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16b261e674b5fc0221acefaf8aa0ed6b1b98af78187d75b09e1006787f60a040/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4916614f-8a34-491d-b4ad-bae72c14b140/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-91 DeviceMajor:0 DeviceMinor:91 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-194 DeviceMajor:0 DeviceMinor:194 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1135b6d0397687f9487e54cfc133c8d6a93b6b9cccf91599c4b75ef7c3cddf85/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9597c7254279a4acc36b251def7dec3cb05d5ecffc4b49fe9fc95a4bcdeb70c7/userdata/shm DeviceMajor:0 DeviceMinor:274 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/33360cd4-4738-4c3d-bc9f-9f5003408589/volumes/kubernetes.io~projected/kube-api-access-fmn55 DeviceMajor:0 DeviceMinor:65 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1/volumes/kubernetes.io~projected/kube-api-access-45m2j DeviceMajor:0 DeviceMinor:69 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a32d691bc1c503e421d0c7cade812b5ed084aa8b5ffcda1cfa550ec243d2c1e4/userdata/shm DeviceMajor:0 DeviceMinor:106 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ea8559a-aa64-4d00-84c0-2c9405e32606/volumes/kubernetes.io~projected/kube-api-access-klzkm DeviceMajor:0 DeviceMinor:70 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dab4a141-b01a-4434-a792-d8fc9d63a83d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:218 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-113 DeviceMajor:0 DeviceMinor:113 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-199 DeviceMajor:0 DeviceMinor:199 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4026e493-67d9-4b2c-8267-1da3a0b1d621/volumes/kubernetes.io~projected/kube-api-access-hplpk DeviceMajor:0 DeviceMinor:219 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1ae4b768-8a6b-4c10-ae02-c1a3950d830c/volumes/kubernetes.io~projected/kube-api-access-hhdc6 DeviceMajor:0 DeviceMinor:221 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-121 DeviceMajor:0 DeviceMinor:121 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-123 DeviceMajor:0 DeviceMinor:123 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0061f2cf-c501-47b0-a89b-3e244f9c313d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/be01642a04bd5f86f0f4682b268dea238129b34cadf8482f8ab0c03cdf7f27b5/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/c147a7be-d6f6-458e-8b0d-70da7a3c7026/volumes/kubernetes.io~projected/kube-api-access-qb8pj DeviceMajor:0 DeviceMinor:68 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-189 DeviceMajor:0 DeviceMinor:189 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9df89bff6b79e8cfaddaf04603087a596c31dbbb8591f9c9d0abb9e16da76f5d/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8258a9c17fdfce96ec8fdaa0b7d8b53e69dd050c3f578c21279eca26a2ded146/userdata/shm DeviceMajor:0 DeviceMinor:99 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1ae4b768-8a6b-4c10-ae02-c1a3950d830c/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/f99954e5-6dfe-457f-9a8f-22752798bb5d/volumes/kubernetes.io~projected/kube-api-access-hsddk DeviceMajor:0 DeviceMinor:61 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/256d11bdaba8884eeb5c071b63c56ee8fbe9b4f8b79ca262d8b6727b9d4e3353/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-87 DeviceMajor:0 DeviceMinor:87 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8ad4438-da5a-4f3a-bb5b-07c330aef792/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:263 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/43eb1690a2264835c978f4dcc58bf4cfc8c17d25c0ff29a4f73e3f8922099f95/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/6ea8559a-aa64-4d00-84c0-2c9405e32606/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:60 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5738903c517750fdd3189dd5853f08e305912b8e74f0504ec99dde0ffbb5d401/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59ab3ed9c9fb828acdc99da907d98147baa0b8793345fd3d6c9f8e932ef21779/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/fd35650d-2c53-4230-8518-c4cb05b59558/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:268 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99a7b227-aad8-414f-8668-70a5660d6a6e/volumes/kubernetes.io~projected/kube-api-access-qvp94 DeviceMajor:0 DeviceMinor:62 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a8ad4438-da5a-4f3a-bb5b-07c330aef792/volumes/kubernetes.io~projected/kube-api-access-x4845 DeviceMajor:0 DeviceMinor:66 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4916614f-8a34-491d-b4ad-bae72c14b140/volumes/kubernetes.io~projected/kube-api-access-6d5dd DeviceMajor:0 DeviceMinor:67 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51fcfffe-f89d-4064-a0e3-45461cc62570/volumes/kubernetes.io~projected/kube-api-access-wd8z9 DeviceMajor:0 DeviceMinor:64 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1135b6d0397687f MacAddress:62:e4:48:96:58:59 Speed:10000 Mtu:8901} {Name:16891bc02d31086 MacAddress:8e:75:bf:3b:92:ff Speed:10000 Mtu:8901} {Name:59ab3ed9c9fb828 MacAddress:fa:bb:88:05:15:c2 Speed:10000 Mtu:8901} {Name:9597c7254279a4a MacAddress:8e:f1:4f:d3:aa:89 Speed:10000 Mtu:8901} {Name:addbe92aa928a46 MacAddress:52:c7:86:e1:21:d4 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f6:c4:ef:b9 Speed:0 Mtu:9001} {Name:br-int MacAddress:6a:bc:44:ad:8c:80 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:f6:c4:ef:b9 Speed:0 Mtu:9001} {Name:f814ce24c2e1df8 MacAddress:f6:fe:b6:09:26:58 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:3e:dd:19:53:94:57 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:36:2c:d6:55:fd:dc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 29 22:42:26.119542 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.117696 4821 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 29 22:42:26.119542 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.117738 4821 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 29 22:42:26.119542 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.118033 4821 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 29 22:42:26.119542 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.118056 4821 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-252.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 29 22:42:26.119542 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.118174 4821 topology_manager.go:143] "Creating topology manager with none policy" Oct 29 22:42:26.119542 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.118182 4821 container_manager_linux.go:311] "Creating device plugin manager" Oct 29 22:42:26.119542 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.118192 4821 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:42:26.119542 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.118231 4821 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:42:26.119542 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.118323 4821 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 29 22:42:26.119542 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.118484 4821 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 29 22:42:26.119542 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.118599 4821 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 29 22:42:26.119542 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.118636 4821 kubelet.go:493] "Attempting to sync node with API server" Oct 29 22:42:26.119542 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.118649 4821 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 29 22:42:26.119542 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.118659 4821 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 29 22:42:26.119542 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.118668 4821 kubelet.go:405] "Adding apiserver pod source" Oct 29 22:42:26.119542 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.118675 4821 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 29 22:42:26.119542 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.119267 4821 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 29 22:42:26.119542 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.119277 4821 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:42:26.120306 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.119836 4821 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 29 22:42:26.122694 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.122680 4821 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 29 22:42:26.122731 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.122695 4821 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:42:26.122803 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.122793 4821 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 29 22:42:26.122981 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.122971 4821 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 29 22:42:26.123014 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.123000 4821 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 29 22:42:26.123094 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.123085 4821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 29 22:42:26.123126 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.123099 4821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 29 22:42:26.123126 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.123109 4821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 29 22:42:26.123126 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.123117 4821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 29 22:42:26.123126 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.123125 4821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 29 22:42:26.123254 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.123131 4821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 29 22:42:26.123254 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.123137 4821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 29 22:42:26.123254 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.123143 4821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 29 22:42:26.123254 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.123150 4821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 29 22:42:26.123254 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.123156 4821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 29 22:42:26.123254 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.123175 4821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 29 22:42:26.123254 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.123186 4821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 29 22:42:26.123254 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.123233 4821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 29 22:42:26.123254 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.123242 4821 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 29 22:42:26.126609 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.126594 4821 server.go:1263] "Started kubelet" Oct 29 22:42:26.126722 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.126671 4821 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 29 22:42:26.127112 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.127050 4821 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 29 22:42:26.127246 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.127134 4821 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 29 22:42:26.127210 ip-10-0-130-252 systemd[1]: Started Kubernetes Kubelet. Oct 29 22:42:26.127486 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.127467 4821 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 29 22:42:26.128096 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.128080 4821 server.go:317] "Adding debug handlers to kubelet server" Oct 29 22:42:26.131545 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.131523 4821 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 29 22:42:26.131687 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:42:26.131657 4821 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-252.ec2.internal\" not found" Oct 29 22:42:26.131687 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.131679 4821 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 29 22:42:26.131912 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.131696 4821 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 29 22:42:26.131912 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.131896 4821 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 29 22:42:26.132230 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.132185 4821 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 29 22:42:26.136577 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.132888 4821 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:36:48 +0000 UTC" deadline="2028-04-15 20:06:21.578658153 +0000 UTC" Oct 29 22:42:26.136577 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.132980 4821 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12813h23m55.445696767s" Oct 29 22:42:26.136577 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.134576 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ea8559a-aa64-4d00-84c0-2c9405e32606" volumeName="kubernetes.io/configmap/6ea8559a-aa64-4d00-84c0-2c9405e32606-env-overrides" seLinuxMountContext="" Oct 29 22:42:26.136577 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.134634 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ea8559a-aa64-4d00-84c0-2c9405e32606" volumeName="kubernetes.io/configmap/6ea8559a-aa64-4d00-84c0-2c9405e32606-ovnkube-config" seLinuxMountContext="" Oct 29 22:42:26.136577 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.134648 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff" volumeName="kubernetes.io/projected/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff-kube-api-access-f6ll2" seLinuxMountContext="" Oct 29 22:42:26.136577 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.134663 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c147a7be-d6f6-458e-8b0d-70da7a3c7026" volumeName="kubernetes.io/configmap/c147a7be-d6f6-458e-8b0d-70da7a3c7026-cni-binary-copy" seLinuxMountContext="" Oct 29 22:42:26.136577 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.134718 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd35650d-2c53-4230-8518-c4cb05b59558" volumeName="kubernetes.io/secret/fd35650d-2c53-4230-8518-c4cb05b59558-original-pull-secret" seLinuxMountContext="" Oct 29 22:42:26.136577 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.134758 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ae4b768-8a6b-4c10-ae02-c1a3950d830c" volumeName="kubernetes.io/secret/1ae4b768-8a6b-4c10-ae02-c1a3950d830c-metrics-tls" seLinuxMountContext="" Oct 29 22:42:26.136577 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.134779 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51fcfffe-f89d-4064-a0e3-45461cc62570" volumeName="kubernetes.io/configmap/51fcfffe-f89d-4064-a0e3-45461cc62570-cni-binary-copy" seLinuxMountContext="" Oct 29 22:42:26.136577 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.134792 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ea8559a-aa64-4d00-84c0-2c9405e32606" volumeName="kubernetes.io/secret/6ea8559a-aa64-4d00-84c0-2c9405e32606-ovn-node-metrics-cert" seLinuxMountContext="" Oct 29 22:42:26.136577 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.134824 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1" volumeName="kubernetes.io/empty-dir/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-tmp" seLinuxMountContext="" Oct 29 22:42:26.136577 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.134841 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f99954e5-6dfe-457f-9a8f-22752798bb5d" volumeName="kubernetes.io/projected/f99954e5-6dfe-457f-9a8f-22752798bb5d-kube-api-access-hsddk" seLinuxMountContext="" Oct 29 22:42:26.136577 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.134878 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33360cd4-4738-4c3d-bc9f-9f5003408589" volumeName="kubernetes.io/empty-dir/33360cd4-4738-4c3d-bc9f-9f5003408589-tmp-dir" seLinuxMountContext="" Oct 29 22:42:26.136577 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.134896 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4026e493-67d9-4b2c-8267-1da3a0b1d621" volumeName="kubernetes.io/secret/4026e493-67d9-4b2c-8267-1da3a0b1d621-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 29 22:42:26.136577 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.134911 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4916614f-8a34-491d-b4ad-bae72c14b140" volumeName="kubernetes.io/projected/4916614f-8a34-491d-b4ad-bae72c14b140-kube-api-access-6d5dd" seLinuxMountContext="" Oct 29 22:42:26.136577 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.134951 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51fcfffe-f89d-4064-a0e3-45461cc62570" volumeName="kubernetes.io/configmap/51fcfffe-f89d-4064-a0e3-45461cc62570-multus-daemon-config" seLinuxMountContext="" Oct 29 22:42:26.136577 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.134991 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99a7b227-aad8-414f-8668-70a5660d6a6e" volumeName="kubernetes.io/projected/99a7b227-aad8-414f-8668-70a5660d6a6e-kube-api-access-qvp94" seLinuxMountContext="" Oct 29 22:42:26.136577 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.135013 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ea8559a-aa64-4d00-84c0-2c9405e32606" volumeName="kubernetes.io/configmap/6ea8559a-aa64-4d00-84c0-2c9405e32606-ovnkube-script-lib" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137079 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af139c4a-79a2-47b4-95e4-493e3430be98" volumeName="kubernetes.io/projected/af139c4a-79a2-47b4-95e4-493e3430be98-kube-api-access-9w9d9" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137111 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dab4a141-b01a-4434-a792-d8fc9d63a83d" volumeName="kubernetes.io/secret/dab4a141-b01a-4434-a792-d8fc9d63a83d-cert" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137127 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f99954e5-6dfe-457f-9a8f-22752798bb5d" volumeName="kubernetes.io/configmap/f99954e5-6dfe-457f-9a8f-22752798bb5d-serviceca" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137140 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ae4b768-8a6b-4c10-ae02-c1a3950d830c" volumeName="kubernetes.io/projected/1ae4b768-8a6b-4c10-ae02-c1a3950d830c-kube-api-access-hhdc6" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137152 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4026e493-67d9-4b2c-8267-1da3a0b1d621" volumeName="kubernetes.io/configmap/4026e493-67d9-4b2c-8267-1da3a0b1d621-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137164 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4916614f-8a34-491d-b4ad-bae72c14b140" volumeName="kubernetes.io/secret/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137176 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ea8559a-aa64-4d00-84c0-2c9405e32606" volumeName="kubernetes.io/projected/6ea8559a-aa64-4d00-84c0-2c9405e32606-kube-api-access-klzkm" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137188 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99a7b227-aad8-414f-8668-70a5660d6a6e" volumeName="kubernetes.io/configmap/99a7b227-aad8-414f-8668-70a5660d6a6e-iptables-alerter-script" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137217 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c147a7be-d6f6-458e-8b0d-70da7a3c7026" volumeName="kubernetes.io/configmap/c147a7be-d6f6-458e-8b0d-70da7a3c7026-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137229 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c147a7be-d6f6-458e-8b0d-70da7a3c7026" volumeName="kubernetes.io/configmap/c147a7be-d6f6-458e-8b0d-70da7a3c7026-cni-sysctl-allowlist" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137242 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dab4a141-b01a-4434-a792-d8fc9d63a83d" volumeName="kubernetes.io/projected/dab4a141-b01a-4434-a792-d8fc9d63a83d-kube-api-access-k7tgw" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137254 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ae4b768-8a6b-4c10-ae02-c1a3950d830c" volumeName="kubernetes.io/configmap/1ae4b768-8a6b-4c10-ae02-c1a3950d830c-config-volume" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137272 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4026e493-67d9-4b2c-8267-1da3a0b1d621" volumeName="kubernetes.io/projected/4026e493-67d9-4b2c-8267-1da3a0b1d621-kube-api-access-hplpk" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137284 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8ad4438-da5a-4f3a-bb5b-07c330aef792" volumeName="kubernetes.io/secret/a8ad4438-da5a-4f3a-bb5b-07c330aef792-metrics-certs" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137296 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c147a7be-d6f6-458e-8b0d-70da7a3c7026" volumeName="kubernetes.io/projected/c147a7be-d6f6-458e-8b0d-70da7a3c7026-kube-api-access-qb8pj" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137309 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1" volumeName="kubernetes.io/projected/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-kube-api-access-45m2j" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137348 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0061f2cf-c501-47b0-a89b-3e244f9c313d" volumeName="kubernetes.io/configmap/0061f2cf-c501-47b0-a89b-3e244f9c313d-konnectivity-ca" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137360 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33360cd4-4738-4c3d-bc9f-9f5003408589" volumeName="kubernetes.io/projected/33360cd4-4738-4c3d-bc9f-9f5003408589-kube-api-access-fmn55" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137373 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4916614f-8a34-491d-b4ad-bae72c14b140" volumeName="kubernetes.io/configmap/4916614f-8a34-491d-b4ad-bae72c14b140-metrics-client-ca" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137389 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4916614f-8a34-491d-b4ad-bae72c14b140" volumeName="kubernetes.io/empty-dir/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-textfile" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137400 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4916614f-8a34-491d-b4ad-bae72c14b140" volumeName="kubernetes.io/secret/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-tls" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137413 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8ad4438-da5a-4f3a-bb5b-07c330aef792" volumeName="kubernetes.io/projected/a8ad4438-da5a-4f3a-bb5b-07c330aef792-kube-api-access-x4845" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137425 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1" volumeName="kubernetes.io/empty-dir/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-tuned" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137436 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0061f2cf-c501-47b0-a89b-3e244f9c313d" volumeName="kubernetes.io/secret/0061f2cf-c501-47b0-a89b-3e244f9c313d-agent-certs" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137448 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4026e493-67d9-4b2c-8267-1da3a0b1d621" volumeName="kubernetes.io/empty-dir/4026e493-67d9-4b2c-8267-1da3a0b1d621-data-volume" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137462 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4916614f-8a34-491d-b4ad-bae72c14b140" volumeName="kubernetes.io/configmap/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137474 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ae4b768-8a6b-4c10-ae02-c1a3950d830c" volumeName="kubernetes.io/empty-dir/1ae4b768-8a6b-4c10-ae02-c1a3950d830c-tmp-dir" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137485 4821 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51fcfffe-f89d-4064-a0e3-45461cc62570" volumeName="kubernetes.io/projected/51fcfffe-f89d-4064-a0e3-45461cc62570-kube-api-access-wd8z9" seLinuxMountContext="" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137497 4821 reconstruct.go:97] "Volume reconstruction finished" Oct 29 22:42:26.137492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137503 4821 reconciler.go:29] "Reconciler: start to sync state" Oct 29 22:42:26.138592 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137554 4821 factory.go:153] Registering CRI-O factory Oct 29 22:42:26.138592 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137572 4821 factory.go:223] Registration of the crio container factory successfully Oct 29 22:42:26.138592 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137630 4821 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 29 22:42:26.138592 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137641 4821 factory.go:55] Registering systemd factory Oct 29 22:42:26.138592 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137650 4821 factory.go:223] Registration of the systemd container factory successfully Oct 29 22:42:26.138592 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137669 4821 factory.go:103] Registering Raw factory Oct 29 22:42:26.138592 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.137685 4821 manager.go:1199] Started watching for new ooms in manager Oct 29 22:42:26.138592 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.138045 4821 manager.go:319] Starting recovery of all containers Oct 29 22:42:26.159971 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.159935 4821 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 29 22:42:26.161726 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.161702 4821 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 29 22:42:26.161726 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.161728 4821 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 29 22:42:26.161852 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.161800 4821 kubelet.go:2521] "Starting kubelet main sync loop" Oct 29 22:42:26.161899 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:42:26.161842 4821 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 29 22:42:26.176231 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.176176 4821 generic.go:356] "Generic (PLEG): container finished" podID="4916614f-8a34-491d-b4ad-bae72c14b140" containerID="a0b917804dc11cc0b5b3024c5019a3f172efdf89df4089f20c127820ffcaa56b" exitCode=0 Oct 29 22:42:26.180657 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.180632 4821 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 29 22:42:26.181885 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.181866 4821 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:42:26.181985 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.181968 4821 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:42:26.187144 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.187120 4821 generic.go:356] "Generic (PLEG): container finished" podID="3f8a08c2f1d6fb4fc298c3bba521bafd" containerID="bbfd3a9d9702654a11c3b592516fa8f84fdbca5a49d8fc28286ce806bbb0f281" exitCode=0 Oct 29 22:42:26.192070 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.192051 4821 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:42:26.203632 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.203569 4821 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:42:26.210022 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.209998 4821 generic.go:356] "Generic (PLEG): container finished" podID="c147a7be-d6f6-458e-8b0d-70da7a3c7026" containerID="b8a85846a0be05835d90d7d7d2b12aea5e67e49b1ce0189e1049bae1b6e729b8" exitCode=0 Oct 29 22:42:26.210022 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.210018 4821 generic.go:356] "Generic (PLEG): container finished" podID="c147a7be-d6f6-458e-8b0d-70da7a3c7026" containerID="7db4c610461011a360e45bebc5cd9f3a812debacd4793a26bea16edd32ba0f80" exitCode=0 Oct 29 22:42:26.210163 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.210027 4821 generic.go:356] "Generic (PLEG): container finished" podID="c147a7be-d6f6-458e-8b0d-70da7a3c7026" containerID="6467ad9bfa7075c9a0ebfc00d2a1dbf90d197c6210eedb39840dd8881ac9963f" exitCode=0 Oct 29 22:42:26.210163 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.210033 4821 generic.go:356] "Generic (PLEG): container finished" podID="c147a7be-d6f6-458e-8b0d-70da7a3c7026" containerID="c34262dbde983262b88881152c6f97adc24862fc36c4c23c49997febb8237be7" exitCode=0 Oct 29 22:42:26.210163 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.210040 4821 generic.go:356] "Generic (PLEG): container finished" podID="c147a7be-d6f6-458e-8b0d-70da7a3c7026" containerID="1268dd417c60587166b2e56108b0e7223294c5abf3be1f146c24957b332035ae" exitCode=0 Oct 29 22:42:26.210163 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.210045 4821 generic.go:356] "Generic (PLEG): container finished" podID="c147a7be-d6f6-458e-8b0d-70da7a3c7026" containerID="60141d85a7f0048d0ba923db485e1427d138632ff50d9568cebfd52e82f79f20" exitCode=0 Oct 29 22:42:26.220827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.220809 4821 manager.go:324] Recovery completed Oct 29 22:42:26.237308 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.237288 4821 cpu_manager.go:226] "Starting" policy="none" Oct 29 22:42:26.237308 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.237305 4821 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 29 22:42:26.237462 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.237324 4821 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 29 22:42:26.237462 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.237435 4821 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 29 22:42:26.237462 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.237446 4821 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 29 22:42:26.237597 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.237578 4821 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 29 22:42:26.237597 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.237585 4821 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 29 22:42:26.237597 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.237590 4821 policy_none.go:50] "Start" Oct 29 22:42:26.237722 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.237603 4821 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 29 22:42:26.237722 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.237613 4821 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 29 22:42:26.250919 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.250900 4821 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 29 22:42:26.250919 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.250915 4821 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 29 22:42:26.250919 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.250921 4821 policy_none.go:44] "Start" Oct 29 22:42:26.254936 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.254907 4821 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 29 22:42:26.257471 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.257459 4821 manager.go:341] "Starting Device Plugin manager" Oct 29 22:42:26.258353 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:42:26.257484 4821 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 29 22:42:26.258420 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.258364 4821 server.go:87] "Starting device plugin registration server" Oct 29 22:42:26.258726 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.258710 4821 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 29 22:42:26.258784 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.258730 4821 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 29 22:42:26.259052 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.259035 4821 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 29 22:42:26.259165 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.259134 4821 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 29 22:42:26.259253 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.259175 4821 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 29 22:42:26.259253 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.259182 4821 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 29 22:42:26.259361 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.259264 4821 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:42:26.259163973Z","UUID":"c2d06c95-94e3-46f9-bb7d-45229f59271f","Handler":null,"Name":"","Endpoint":""} Oct 29 22:42:26.261371 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:42:26.261346 4821 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 29 22:42:26.261478 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.261369 4821 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 29 22:42:26.261478 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.261394 4821 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 29 22:42:26.262036 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.261996 4821 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-252.ec2.internal"] Oct 29 22:42:26.263101 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.263069 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" event={"ID":"3f8a08c2f1d6fb4fc298c3bba521bafd","Type":"ContainerStarted","Data":"8be65d7a8dfe56c1cdee7f5844d005504615b923195713109b0672866c8fd3b7"} Oct 29 22:42:26.263180 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.263113 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" event={"ID":"3f8a08c2f1d6fb4fc298c3bba521bafd","Type":"ContainerDied","Data":"bbfd3a9d9702654a11c3b592516fa8f84fdbca5a49d8fc28286ce806bbb0f281"} Oct 29 22:42:26.263180 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.263135 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" event={"ID":"3f8a08c2f1d6fb4fc298c3bba521bafd","Type":"ContainerStarted","Data":"43eb1690a2264835c978f4dcc58bf4cfc8c17d25c0ff29a4f73e3f8922099f95"} Oct 29 22:42:26.263314 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.263191 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-252.ec2.internal" event={"ID":"83fee9c1d2770428373c9940b59a21be","Type":"ContainerStarted","Data":"ceda17f91eddb9cc7addb9a8d9bb8710f7cdc2fb336929d04e0fca0571beaf63"} Oct 29 22:42:26.263314 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.263237 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-252.ec2.internal" event={"ID":"83fee9c1d2770428373c9940b59a21be","Type":"ContainerStarted","Data":"16b261e674b5fc0221acefaf8aa0ed6b1b98af78187d75b09e1006787f60a040"} Oct 29 22:42:26.263584 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.263561 4821 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" Oct 29 22:42:26.263675 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.263668 4821 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-252.ec2.internal" Oct 29 22:42:26.288182 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.288153 4821 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:42:26.288336 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:42:26.288231 4821 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-252.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-252.ec2.internal" Oct 29 22:42:26.291382 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.291366 4821 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:42:26.291450 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:42:26.291414 4821 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" Oct 29 22:42:26.359852 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.359816 4821 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:42:26.361240 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.361222 4821 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:42:26.361322 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.361255 4821 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:42:26.361322 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.361265 4821 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-252.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:42:26.361322 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.361299 4821 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-252.ec2.internal" Oct 29 22:42:26.381730 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.381705 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/83fee9c1d2770428373c9940b59a21be-config\") pod \"kube-apiserver-proxy-ip-10-0-130-252.ec2.internal\" (UID: \"83fee9c1d2770428373c9940b59a21be\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-252.ec2.internal" Oct 29 22:42:26.381857 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.381735 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f8a08c2f1d6fb4fc298c3bba521bafd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal\" (UID: \"3f8a08c2f1d6fb4fc298c3bba521bafd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" Oct 29 22:42:26.381857 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.381751 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f8a08c2f1d6fb4fc298c3bba521bafd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal\" (UID: \"3f8a08c2f1d6fb4fc298c3bba521bafd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" Oct 29 22:42:26.394350 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.394325 4821 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-252.ec2.internal" Oct 29 22:42:26.394448 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.394385 4821 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-252.ec2.internal" Oct 29 22:42:26.482323 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.482229 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/83fee9c1d2770428373c9940b59a21be-config\") pod \"kube-apiserver-proxy-ip-10-0-130-252.ec2.internal\" (UID: \"83fee9c1d2770428373c9940b59a21be\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-252.ec2.internal" Oct 29 22:42:26.482323 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.482268 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f8a08c2f1d6fb4fc298c3bba521bafd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal\" (UID: \"3f8a08c2f1d6fb4fc298c3bba521bafd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" Oct 29 22:42:26.482323 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.482288 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f8a08c2f1d6fb4fc298c3bba521bafd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal\" (UID: \"3f8a08c2f1d6fb4fc298c3bba521bafd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" Oct 29 22:42:26.482561 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.482350 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/83fee9c1d2770428373c9940b59a21be-config\") pod \"kube-apiserver-proxy-ip-10-0-130-252.ec2.internal\" (UID: \"83fee9c1d2770428373c9940b59a21be\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-252.ec2.internal" Oct 29 22:42:26.482561 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.482354 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f8a08c2f1d6fb4fc298c3bba521bafd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal\" (UID: \"3f8a08c2f1d6fb4fc298c3bba521bafd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" Oct 29 22:42:26.482561 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:26.482360 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f8a08c2f1d6fb4fc298c3bba521bafd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal\" (UID: \"3f8a08c2f1d6fb4fc298c3bba521bafd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" Oct 29 22:42:27.119170 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.119136 4821 apiserver.go:52] "Watching apiserver" Oct 29 22:42:27.158277 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.158241 4821 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 29 22:42:27.158822 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.158799 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-252.ec2.internal","openshift-ingress-canary/ingress-canary-9g7rk","openshift-multus/network-metrics-daemon-29dzf","kube-system/global-pull-secret-syncer-ldgnp","openshift-image-registry/node-ca-7kgv6","openshift-network-diagnostics/network-check-target-mbv7x","openshift-dns/dns-default-zb2ch","openshift-dns/node-resolver-cw2sv","openshift-cluster-node-tuning-operator/tuned-m4s7g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal","openshift-multus/multus-vdz2c","openshift-ovn-kubernetes/ovnkube-node-9b8jf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8","openshift-multus/multus-additional-cni-plugins-rdtdb","openshift-insights/insights-runtime-extractor-k7c6q","kube-system/konnectivity-agent-jzjsw","openshift-monitoring/node-exporter-shnd6","openshift-network-operator/iptables-alerter-bzvh4"] Oct 29 22:42:27.174836 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.174813 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g9cd8\"" Oct 29 22:42:27.174979 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.174938 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 29 22:42:27.174979 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.174946 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vd45d\"" Oct 29 22:42:27.174979 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.174943 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 29 22:42:27.174979 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.174962 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:42:27.174979 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.174978 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 29 22:42:27.175153 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.174959 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 29 22:42:27.175153 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.174944 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 29 22:42:27.175153 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.174989 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 29 22:42:27.176442 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.176430 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zrmx7\"" Oct 29 22:42:27.184772 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.184749 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p6rt5\"" Oct 29 22:42:27.184932 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.184913 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 22:42:27.185183 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.185168 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f9xcm\"" Oct 29 22:42:27.185599 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.185580 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8l2jp\"" Oct 29 22:42:27.185599 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.185594 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kr4qv\"" Oct 29 22:42:27.185702 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.185588 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mhfn7\"" Oct 29 22:42:27.185702 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.185583 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 29 22:42:27.186186 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.186153 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rgvrg\"" Oct 29 22:42:27.186186 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.186166 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 29 22:42:27.186186 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.186175 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pd9xt\"" Oct 29 22:42:27.186408 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.186222 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 29 22:42:27.186551 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.186536 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c4xfj\"" Oct 29 22:42:27.186651 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.186539 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rfj6j\"" Oct 29 22:42:27.186651 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.186577 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 22:42:27.186736 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.186695 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:42:27.186889 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.186874 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 29 22:42:27.186993 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.186978 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gg4pb\"" Oct 29 22:42:27.187062 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.187035 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 29 22:42:27.187129 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.187115 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-dvjrv\"" Oct 29 22:42:27.187244 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.187174 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-wccb2\"" Oct 29 22:42:27.187333 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.187254 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 29 22:42:27.187586 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.187571 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 29 22:42:27.187586 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.187579 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 29 22:42:27.187674 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.187575 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xzwzj\"" Oct 29 22:42:27.188360 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.188342 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 29 22:42:27.188488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.188360 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 29 22:42:27.188488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.188391 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 29 22:42:27.188488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.188457 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 29 22:42:27.188488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.188476 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 29 22:42:27.188770 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.188468 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 29 22:42:27.188770 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.188631 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 29 22:42:27.188770 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.188706 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 29 22:42:27.188770 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.188711 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 29 22:42:27.188770 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.188720 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 29 22:42:27.189124 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.188808 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 29 22:42:27.189124 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.188832 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 29 22:42:27.189124 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.188868 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 22:42:27.189124 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.188878 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 29 22:42:27.189124 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.188896 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 22:42:27.189124 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.188918 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 29 22:42:27.189124 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.188957 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 29 22:42:27.189124 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.188998 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 29 22:42:27.189124 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.189005 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 29 22:42:27.189124 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.188998 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 29 22:42:27.189124 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.189023 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 29 22:42:27.189124 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.189079 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 29 22:42:27.189124 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.189083 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 29 22:42:27.189124 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.189113 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 29 22:42:27.189612 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.189590 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 29 22:42:27.196428 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.196416 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 29 22:42:27.196602 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.196585 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 29 22:42:27.196662 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.196641 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 29 22:42:27.215565 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.214483 4821 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-252.ec2.internal" Oct 29 22:42:27.215565 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.215032 4821 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" Oct 29 22:42:27.233511 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.233493 4821 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 29 22:42:27.237340 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.237323 4821 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:42:27.237430 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:42:27.237363 4821 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-252.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-252.ec2.internal" Oct 29 22:42:27.240342 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.240324 4821 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:42:27.240437 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:42:27.240364 4821 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-252.ec2.internal" Oct 29 22:42:27.285678 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.285649 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-run-k8s-cni-cncf-io\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.285816 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.285692 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4026e493-67d9-4b2c-8267-1da3a0b1d621-data-volume\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:27.285816 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.285721 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/99a7b227-aad8-414f-8668-70a5660d6a6e-iptables-alerter-script\") pod \"iptables-alerter-bzvh4\" (UID: \"99a7b227-aad8-414f-8668-70a5660d6a6e\") " pod="openshift-network-operator/iptables-alerter-bzvh4" Oct 29 22:42:27.285816 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.285781 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c147a7be-d6f6-458e-8b0d-70da7a3c7026-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:42:27.285999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.285820 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c147a7be-d6f6-458e-8b0d-70da7a3c7026-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:42:27.285999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.285857 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-cnibin\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.285999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.285883 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-var-lib-cni-multus\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.285999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.285901 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4026e493-67d9-4b2c-8267-1da3a0b1d621-data-volume\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:27.285999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.285964 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhdc6\" (UniqueName: \"kubernetes.io/projected/1ae4b768-8a6b-4c10-ae02-c1a3950d830c-kube-api-access-hhdc6\") pod \"dns-default-zb2ch\" (UID: \"1ae4b768-8a6b-4c10-ae02-c1a3950d830c\") " pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:27.286263 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.286001 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4916614f-8a34-491d-b4ad-bae72c14b140-sys\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:42:27.286263 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.286042 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-sysctl-d\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.286263 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.286064 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dab4a141-b01a-4434-a792-d8fc9d63a83d-cert\") pod \"ingress-canary-9g7rk\" (UID: \"dab4a141-b01a-4434-a792-d8fc9d63a83d\") " pod="openshift-ingress-canary/ingress-canary-9g7rk" Oct 29 22:42:27.286263 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.286089 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd8z9\" (UniqueName: \"kubernetes.io/projected/51fcfffe-f89d-4064-a0e3-45461cc62570-kube-api-access-wd8z9\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.286263 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.286134 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c147a7be-d6f6-458e-8b0d-70da7a3c7026-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:42:27.286263 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.286134 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/99a7b227-aad8-414f-8668-70a5660d6a6e-iptables-alerter-script\") pod \"iptables-alerter-bzvh4\" (UID: \"99a7b227-aad8-414f-8668-70a5660d6a6e\") " pod="openshift-network-operator/iptables-alerter-bzvh4" Oct 29 22:42:27.286263 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.286137 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c147a7be-d6f6-458e-8b0d-70da7a3c7026-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:42:27.286263 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.286130 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-etc-openvswitch\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.286263 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.286232 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f99954e5-6dfe-457f-9a8f-22752798bb5d-host\") pod \"node-ca-7kgv6\" (UID: \"f99954e5-6dfe-457f-9a8f-22752798bb5d\") " pod="openshift-image-registry/node-ca-7kgv6" Oct 29 22:42:27.286662 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.286304 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-multus-socket-dir-parent\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.286662 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.286354 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-multus-conf-dir\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.286662 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.286384 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-etc-selinux\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:42:27.286662 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.286414 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klzkm\" (UniqueName: \"kubernetes.io/projected/6ea8559a-aa64-4d00-84c0-2c9405e32606-kube-api-access-klzkm\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.286662 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.286433 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dab4a141-b01a-4434-a792-d8fc9d63a83d-cert\") pod \"ingress-canary-9g7rk\" (UID: \"dab4a141-b01a-4434-a792-d8fc9d63a83d\") " pod="openshift-ingress-canary/ingress-canary-9g7rk" Oct 29 22:42:27.286662 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.286442 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4916614f-8a34-491d-b4ad-bae72c14b140-metrics-client-ca\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:42:27.286662 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.286477 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d5dd\" (UniqueName: \"kubernetes.io/projected/4916614f-8a34-491d-b4ad-bae72c14b140-kube-api-access-6d5dd\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:42:27.286662 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.286573 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-run-netns\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.287014 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.286722 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6ll2\" (UniqueName: \"kubernetes.io/projected/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff-kube-api-access-f6ll2\") pod \"network-check-target-mbv7x\" (UID: \"b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff\") " pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:42:27.287014 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.286758 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4916614f-8a34-491d-b4ad-bae72c14b140-metrics-client-ca\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:42:27.287014 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.286811 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4026e493-67d9-4b2c-8267-1da3a0b1d621-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:27.287014 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.286840 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-run-ovn\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.287014 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.286857 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ea8559a-aa64-4d00-84c0-2c9405e32606-ovnkube-config\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.287014 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.286875 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ea8559a-aa64-4d00-84c0-2c9405e32606-ovn-node-metrics-cert\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.287014 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.286931 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4916614f-8a34-491d-b4ad-bae72c14b140-root\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:42:27.287350 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287034 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-wtmp\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:42:27.287350 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287133 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae4b768-8a6b-4c10-ae02-c1a3950d830c-tmp-dir\") pod \"dns-default-zb2ch\" (UID: \"1ae4b768-8a6b-4c10-ae02-c1a3950d830c\") " pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:27.287350 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287166 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ea8559a-aa64-4d00-84c0-2c9405e32606-ovn-node-metrics-cert\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.287350 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287194 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ea8559a-aa64-4d00-84c0-2c9405e32606-ovnkube-config\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.287350 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287223 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae4b768-8a6b-4c10-ae02-c1a3950d830c-tmp-dir\") pod \"dns-default-zb2ch\" (UID: \"1ae4b768-8a6b-4c10-ae02-c1a3950d830c\") " pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:27.287350 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287186 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-kubelet\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.287350 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287261 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fd35650d-2c53-4230-8518-c4cb05b59558-kubelet-config\") pod \"global-pull-secret-syncer-ldgnp\" (UID: \"fd35650d-2c53-4230-8518-c4cb05b59558\") " pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:42:27.287350 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287264 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4026e493-67d9-4b2c-8267-1da3a0b1d621-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:27.287350 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287278 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c147a7be-d6f6-458e-8b0d-70da7a3c7026-cnibin\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:42:27.287350 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287292 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-kubernetes\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.287350 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287339 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33360cd4-4738-4c3d-bc9f-9f5003408589-hosts-file\") pod \"node-resolver-cw2sv\" (UID: \"33360cd4-4738-4c3d-bc9f-9f5003408589\") " pod="openshift-dns/node-resolver-cw2sv" Oct 29 22:42:27.287842 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287373 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-system-cni-dir\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.287842 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287416 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w9d9\" (UniqueName: \"kubernetes.io/projected/af139c4a-79a2-47b4-95e4-493e3430be98-kube-api-access-9w9d9\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:42:27.287842 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287447 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-run-netns\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.287842 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287468 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-run-systemd\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.287842 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287503 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-cni-bin\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.287842 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287525 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-accelerators-collector-config\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:42:27.287842 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287569 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33360cd4-4738-4c3d-bc9f-9f5003408589-tmp-dir\") pod \"node-resolver-cw2sv\" (UID: \"33360cd4-4738-4c3d-bc9f-9f5003408589\") " pod="openshift-dns/node-resolver-cw2sv" Oct 29 22:42:27.287842 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287603 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:42:27.287842 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287631 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-device-dir\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:42:27.287842 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287695 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33360cd4-4738-4c3d-bc9f-9f5003408589-tmp-dir\") pod \"node-resolver-cw2sv\" (UID: \"33360cd4-4738-4c3d-bc9f-9f5003408589\") " pod="openshift-dns/node-resolver-cw2sv" Oct 29 22:42:27.287842 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287706 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-tls\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:42:27.287842 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287739 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-sysctl-conf\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.287842 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287764 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-accelerators-collector-config\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:42:27.287842 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287778 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-node-log\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.287842 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287815 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fd35650d-2c53-4230-8518-c4cb05b59558-dbus\") pod \"global-pull-secret-syncer-ldgnp\" (UID: \"fd35650d-2c53-4230-8518-c4cb05b59558\") " pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:42:27.288437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287861 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c147a7be-d6f6-458e-8b0d-70da7a3c7026-system-cni-dir\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:42:27.288437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287886 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-run\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.288437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287904 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-tuned\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.288437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287920 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-run-multus-certs\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.288437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287945 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-tls\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:42:27.288437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287959 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-log-socket\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.288437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.287997 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.288437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288004 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-tuned\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.288437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288040 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd35650d-2c53-4230-8518-c4cb05b59558-original-pull-secret\") pod \"global-pull-secret-syncer-ldgnp\" (UID: \"fd35650d-2c53-4230-8518-c4cb05b59558\") " pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:42:27.288437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288138 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-modprobe-d\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.288437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288157 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-sysconfig\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.288437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288179 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-lib-modules\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.288437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288225 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fd35650d-2c53-4230-8518-c4cb05b59558-original-pull-secret\") pod \"global-pull-secret-syncer-ldgnp\" (UID: \"fd35650d-2c53-4230-8518-c4cb05b59558\") " pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:42:27.288437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288234 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-host\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.288437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288272 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f99954e5-6dfe-457f-9a8f-22752798bb5d-serviceca\") pod \"node-ca-7kgv6\" (UID: \"f99954e5-6dfe-457f-9a8f-22752798bb5d\") " pod="openshift-image-registry/node-ca-7kgv6" Oct 29 22:42:27.288437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288303 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51fcfffe-f89d-4064-a0e3-45461cc62570-cni-binary-copy\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.288437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288318 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4026e493-67d9-4b2c-8267-1da3a0b1d621-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:27.288437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288335 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99a7b227-aad8-414f-8668-70a5660d6a6e-host-slash\") pod \"iptables-alerter-bzvh4\" (UID: \"99a7b227-aad8-414f-8668-70a5660d6a6e\") " pod="openshift-network-operator/iptables-alerter-bzvh4" Oct 29 22:42:27.288437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288351 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvp94\" (UniqueName: \"kubernetes.io/projected/99a7b227-aad8-414f-8668-70a5660d6a6e-kube-api-access-qvp94\") pod \"iptables-alerter-bzvh4\" (UID: \"99a7b227-aad8-414f-8668-70a5660d6a6e\") " pod="openshift-network-operator/iptables-alerter-bzvh4" Oct 29 22:42:27.288437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288368 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-run-ovn-kubernetes\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.288437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288391 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ea8559a-aa64-4d00-84c0-2c9405e32606-env-overrides\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.288437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288428 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb8pj\" (UniqueName: \"kubernetes.io/projected/c147a7be-d6f6-458e-8b0d-70da7a3c7026-kube-api-access-qb8pj\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288457 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-var-lib-kubelet\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288499 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f99954e5-6dfe-457f-9a8f-22752798bb5d-serviceca\") pod \"node-ca-7kgv6\" (UID: \"f99954e5-6dfe-457f-9a8f-22752798bb5d\") " pod="openshift-image-registry/node-ca-7kgv6" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288540 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-run-openvswitch\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288555 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4026e493-67d9-4b2c-8267-1da3a0b1d621-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288569 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ea8559a-aa64-4d00-84c0-2c9405e32606-ovnkube-script-lib\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288595 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/51fcfffe-f89d-4064-a0e3-45461cc62570-cni-binary-copy\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288606 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c147a7be-d6f6-458e-8b0d-70da7a3c7026-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288640 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ea8559a-aa64-4d00-84c0-2c9405e32606-env-overrides\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288754 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7tgw\" (UniqueName: \"kubernetes.io/projected/dab4a141-b01a-4434-a792-d8fc9d63a83d-kube-api-access-k7tgw\") pod \"ingress-canary-9g7rk\" (UID: \"dab4a141-b01a-4434-a792-d8fc9d63a83d\") " pod="openshift-ingress-canary/ingress-canary-9g7rk" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288767 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ea8559a-aa64-4d00-84c0-2c9405e32606-ovnkube-script-lib\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288786 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4845\" (UniqueName: \"kubernetes.io/projected/a8ad4438-da5a-4f3a-bb5b-07c330aef792-kube-api-access-x4845\") pod \"network-metrics-daemon-29dzf\" (UID: \"a8ad4438-da5a-4f3a-bb5b-07c330aef792\") " pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288814 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-multus-cni-dir\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288842 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-hostroot\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288902 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ae4b768-8a6b-4c10-ae02-c1a3950d830c-metrics-tls\") pod \"dns-default-zb2ch\" (UID: \"1ae4b768-8a6b-4c10-ae02-c1a3950d830c\") " pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288930 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4026e493-67d9-4b2c-8267-1da3a0b1d621-crio-socket\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288962 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-registration-dir\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.288989 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-var-lib-openvswitch\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289021 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-cni-netd\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289057 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45m2j\" (UniqueName: \"kubernetes.io/projected/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-kube-api-access-45m2j\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289089 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hplpk\" (UniqueName: \"kubernetes.io/projected/4026e493-67d9-4b2c-8267-1da3a0b1d621-kube-api-access-hplpk\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289119 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1ae4b768-8a6b-4c10-ae02-c1a3950d830c-metrics-tls\") pod \"dns-default-zb2ch\" (UID: \"1ae4b768-8a6b-4c10-ae02-c1a3950d830c\") " pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289142 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-socket-dir\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289253 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-sys-fs\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289283 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c147a7be-d6f6-458e-8b0d-70da7a3c7026-cni-binary-copy\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289322 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-systemd\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289355 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsddk\" (UniqueName: \"kubernetes.io/projected/f99954e5-6dfe-457f-9a8f-22752798bb5d-kube-api-access-hsddk\") pod \"node-ca-7kgv6\" (UID: \"f99954e5-6dfe-457f-9a8f-22752798bb5d\") " pod="openshift-image-registry/node-ca-7kgv6" Oct 29 22:42:27.290299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289382 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-var-lib-cni-bin\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.290299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289408 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-etc-kubernetes\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.290299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289433 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae4b768-8a6b-4c10-ae02-c1a3950d830c-config-volume\") pod \"dns-default-zb2ch\" (UID: \"1ae4b768-8a6b-4c10-ae02-c1a3950d830c\") " pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:27.290299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289472 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-systemd-units\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.290299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289483 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c147a7be-d6f6-458e-8b0d-70da7a3c7026-cni-binary-copy\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:42:27.290299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289529 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:42:27.290299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289645 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-tmp\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.290299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289651 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae4b768-8a6b-4c10-ae02-c1a3950d830c-config-volume\") pod \"dns-default-zb2ch\" (UID: \"1ae4b768-8a6b-4c10-ae02-c1a3950d830c\") " pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:27.290299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289712 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmn55\" (UniqueName: \"kubernetes.io/projected/33360cd4-4738-4c3d-bc9f-9f5003408589-kube-api-access-fmn55\") pod \"node-resolver-cw2sv\" (UID: \"33360cd4-4738-4c3d-bc9f-9f5003408589\") " pod="openshift-dns/node-resolver-cw2sv" Oct 29 22:42:27.290299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289720 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-tmp\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.290299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289752 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51fcfffe-f89d-4064-a0e3-45461cc62570-multus-daemon-config\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.290299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289761 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:42:27.290299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289769 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-slash\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.290299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289803 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-textfile\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:42:27.290299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289908 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-sys\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.290299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289924 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/51fcfffe-f89d-4064-a0e3-45461cc62570-multus-daemon-config\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.290299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289936 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-var-lib-kubelet\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.290299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289943 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-textfile\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:42:27.290299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.289976 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8ad4438-da5a-4f3a-bb5b-07c330aef792-metrics-certs\") pod \"network-metrics-daemon-29dzf\" (UID: \"a8ad4438-da5a-4f3a-bb5b-07c330aef792\") " pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:42:27.290299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.290005 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0061f2cf-c501-47b0-a89b-3e244f9c313d-agent-certs\") pod \"konnectivity-agent-jzjsw\" (UID: \"0061f2cf-c501-47b0-a89b-3e244f9c313d\") " pod="kube-system/konnectivity-agent-jzjsw" Oct 29 22:42:27.290299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.290042 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0061f2cf-c501-47b0-a89b-3e244f9c313d-konnectivity-ca\") pod \"konnectivity-agent-jzjsw\" (UID: \"0061f2cf-c501-47b0-a89b-3e244f9c313d\") " pod="kube-system/konnectivity-agent-jzjsw" Oct 29 22:42:27.290299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.290149 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8ad4438-da5a-4f3a-bb5b-07c330aef792-metrics-certs\") pod \"network-metrics-daemon-29dzf\" (UID: \"a8ad4438-da5a-4f3a-bb5b-07c330aef792\") " pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:42:27.290867 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.290327 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0061f2cf-c501-47b0-a89b-3e244f9c313d-agent-certs\") pod \"konnectivity-agent-jzjsw\" (UID: \"0061f2cf-c501-47b0-a89b-3e244f9c313d\") " pod="kube-system/konnectivity-agent-jzjsw" Oct 29 22:42:27.290867 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.290341 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0061f2cf-c501-47b0-a89b-3e244f9c313d-konnectivity-ca\") pod \"konnectivity-agent-jzjsw\" (UID: \"0061f2cf-c501-47b0-a89b-3e244f9c313d\") " pod="kube-system/konnectivity-agent-jzjsw" Oct 29 22:42:27.308980 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.308953 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd8z9\" (UniqueName: \"kubernetes.io/projected/51fcfffe-f89d-4064-a0e3-45461cc62570-kube-api-access-wd8z9\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.309228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.309191 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6ll2\" (UniqueName: \"kubernetes.io/projected/b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff-kube-api-access-f6ll2\") pod \"network-check-target-mbv7x\" (UID: \"b5f1f5a0-d3e4-4876-8a15-68fe2ff0ddff\") " pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:42:27.314135 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.314101 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvp94\" (UniqueName: \"kubernetes.io/projected/99a7b227-aad8-414f-8668-70a5660d6a6e-kube-api-access-qvp94\") pod \"iptables-alerter-bzvh4\" (UID: \"99a7b227-aad8-414f-8668-70a5660d6a6e\") " pod="openshift-network-operator/iptables-alerter-bzvh4" Oct 29 22:42:27.314135 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.314128 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhdc6\" (UniqueName: \"kubernetes.io/projected/1ae4b768-8a6b-4c10-ae02-c1a3950d830c-kube-api-access-hhdc6\") pod \"dns-default-zb2ch\" (UID: \"1ae4b768-8a6b-4c10-ae02-c1a3950d830c\") " pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:27.314316 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.314278 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d5dd\" (UniqueName: \"kubernetes.io/projected/4916614f-8a34-491d-b4ad-bae72c14b140-kube-api-access-6d5dd\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:42:27.314316 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.314287 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w9d9\" (UniqueName: \"kubernetes.io/projected/af139c4a-79a2-47b4-95e4-493e3430be98-kube-api-access-9w9d9\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:42:27.314391 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.314329 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klzkm\" (UniqueName: \"kubernetes.io/projected/6ea8559a-aa64-4d00-84c0-2c9405e32606-kube-api-access-klzkm\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.314461 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.314445 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7tgw\" (UniqueName: \"kubernetes.io/projected/dab4a141-b01a-4434-a792-d8fc9d63a83d-kube-api-access-k7tgw\") pod \"ingress-canary-9g7rk\" (UID: \"dab4a141-b01a-4434-a792-d8fc9d63a83d\") " pod="openshift-ingress-canary/ingress-canary-9g7rk" Oct 29 22:42:27.315807 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.315785 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hplpk\" (UniqueName: \"kubernetes.io/projected/4026e493-67d9-4b2c-8267-1da3a0b1d621-kube-api-access-hplpk\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:27.315885 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.315855 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmn55\" (UniqueName: \"kubernetes.io/projected/33360cd4-4738-4c3d-bc9f-9f5003408589-kube-api-access-fmn55\") pod \"node-resolver-cw2sv\" (UID: \"33360cd4-4738-4c3d-bc9f-9f5003408589\") " pod="openshift-dns/node-resolver-cw2sv" Oct 29 22:42:27.315941 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.315923 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4845\" (UniqueName: \"kubernetes.io/projected/a8ad4438-da5a-4f3a-bb5b-07c330aef792-kube-api-access-x4845\") pod \"network-metrics-daemon-29dzf\" (UID: \"a8ad4438-da5a-4f3a-bb5b-07c330aef792\") " pod="openshift-multus/network-metrics-daemon-29dzf" Oct 29 22:42:27.315972 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.315925 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsddk\" (UniqueName: \"kubernetes.io/projected/f99954e5-6dfe-457f-9a8f-22752798bb5d-kube-api-access-hsddk\") pod \"node-ca-7kgv6\" (UID: \"f99954e5-6dfe-457f-9a8f-22752798bb5d\") " pod="openshift-image-registry/node-ca-7kgv6" Oct 29 22:42:27.318987 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.318970 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb8pj\" (UniqueName: \"kubernetes.io/projected/c147a7be-d6f6-458e-8b0d-70da7a3c7026-kube-api-access-qb8pj\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:42:27.319054 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.319039 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45m2j\" (UniqueName: \"kubernetes.io/projected/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-kube-api-access-45m2j\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.390723 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.390646 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-multus-cni-dir\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.390723 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.390679 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-hostroot\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.390723 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.390696 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4026e493-67d9-4b2c-8267-1da3a0b1d621-crio-socket\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:27.390723 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.390714 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-registration-dir\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:42:27.390999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.390746 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-var-lib-openvswitch\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.390999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.390757 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-multus-cni-dir\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.390999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.390768 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-cni-netd\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.390999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.390785 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-hostroot\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.390999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.390807 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-socket-dir\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:42:27.390999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.390813 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4026e493-67d9-4b2c-8267-1da3a0b1d621-crio-socket\") pod \"insights-runtime-extractor-k7c6q\" (UID: \"4026e493-67d9-4b2c-8267-1da3a0b1d621\") " pod="openshift-insights/insights-runtime-extractor-k7c6q" Oct 29 22:42:27.390999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.390833 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-sys-fs\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:42:27.390999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.390805 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-cni-netd\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.390999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.390857 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-systemd\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.390999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.390857 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-var-lib-openvswitch\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.390999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.390832 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-registration-dir\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:42:27.390999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.390872 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-sys-fs\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:42:27.390999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.390881 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-var-lib-cni-bin\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.390999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.390920 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-var-lib-cni-bin\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.390999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.390925 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-etc-kubernetes\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.390999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.390897 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-systemd\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.390999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.390953 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-systemd-units\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.390999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.390882 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-socket-dir\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:42:27.390999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.390980 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-systemd-units\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.390999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391000 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-slash\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391013 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-etc-kubernetes\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391022 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-sys\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391049 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-var-lib-kubelet\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391054 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-slash\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391076 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-run-k8s-cni-cncf-io\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391083 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-sys\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391103 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-cnibin\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391126 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-var-lib-cni-multus\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391149 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4916614f-8a34-491d-b4ad-bae72c14b140-sys\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391148 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-run-k8s-cni-cncf-io\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391171 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-var-lib-kubelet\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391176 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-cnibin\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391182 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4916614f-8a34-491d-b4ad-bae72c14b140-sys\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391185 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-var-lib-cni-multus\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391225 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-sysctl-d\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391247 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-etc-openvswitch\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391263 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f99954e5-6dfe-457f-9a8f-22752798bb5d-host\") pod \"node-ca-7kgv6\" (UID: \"f99954e5-6dfe-457f-9a8f-22752798bb5d\") " pod="openshift-image-registry/node-ca-7kgv6" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391279 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-multus-socket-dir-parent\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391300 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-multus-conf-dir\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391298 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-etc-openvswitch\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391315 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-etc-selinux\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391332 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-run-netns\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391348 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-multus-conf-dir\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391375 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-run-ovn\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391397 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-sysctl-d\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391314 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f99954e5-6dfe-457f-9a8f-22752798bb5d-host\") pod \"node-ca-7kgv6\" (UID: \"f99954e5-6dfe-457f-9a8f-22752798bb5d\") " pod="openshift-image-registry/node-ca-7kgv6" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391412 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-etc-selinux\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391409 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-run-netns\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391410 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4916614f-8a34-491d-b4ad-bae72c14b140-root\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391446 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-run-ovn\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391449 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4916614f-8a34-491d-b4ad-bae72c14b140-root\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391449 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-wtmp\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391495 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-kubelet\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391376 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-multus-socket-dir-parent\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391525 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4916614f-8a34-491d-b4ad-bae72c14b140-node-exporter-wtmp\") pod \"node-exporter-shnd6\" (UID: \"4916614f-8a34-491d-b4ad-bae72c14b140\") " pod="openshift-monitoring/node-exporter-shnd6" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391538 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-kubelet\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391575 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fd35650d-2c53-4230-8518-c4cb05b59558-kubelet-config\") pod \"global-pull-secret-syncer-ldgnp\" (UID: \"fd35650d-2c53-4230-8518-c4cb05b59558\") " pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391602 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c147a7be-d6f6-458e-8b0d-70da7a3c7026-cnibin\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391625 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-kubernetes\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391624 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fd35650d-2c53-4230-8518-c4cb05b59558-kubelet-config\") pod \"global-pull-secret-syncer-ldgnp\" (UID: \"fd35650d-2c53-4230-8518-c4cb05b59558\") " pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391649 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33360cd4-4738-4c3d-bc9f-9f5003408589-hosts-file\") pod \"node-resolver-cw2sv\" (UID: \"33360cd4-4738-4c3d-bc9f-9f5003408589\") " pod="openshift-dns/node-resolver-cw2sv" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391666 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-kubernetes\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391674 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-system-cni-dir\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391684 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33360cd4-4738-4c3d-bc9f-9f5003408589-hosts-file\") pod \"node-resolver-cw2sv\" (UID: \"33360cd4-4738-4c3d-bc9f-9f5003408589\") " pod="openshift-dns/node-resolver-cw2sv" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391628 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c147a7be-d6f6-458e-8b0d-70da7a3c7026-cnibin\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391701 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-run-netns\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391717 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-system-cni-dir\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391721 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-run-netns\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391740 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-run-systemd\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391764 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-cni-bin\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391788 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391807 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-run-systemd\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391826 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-cni-bin\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391840 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391854 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-device-dir\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391882 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-sysctl-conf\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391904 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af139c4a-79a2-47b4-95e4-493e3430be98-device-dir\") pod \"aws-ebs-csi-driver-node-tx6m8\" (UID: \"af139c4a-79a2-47b4-95e4-493e3430be98\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tx6m8" Oct 29 22:42:27.391892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391904 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-node-log\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391930 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-node-log\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391946 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fd35650d-2c53-4230-8518-c4cb05b59558-dbus\") pod \"global-pull-secret-syncer-ldgnp\" (UID: \"fd35650d-2c53-4230-8518-c4cb05b59558\") " pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391963 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c147a7be-d6f6-458e-8b0d-70da7a3c7026-system-cni-dir\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391977 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-run\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.391998 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-run-multus-certs\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392012 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-sysctl-conf\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392023 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-run-multus-certs\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392024 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c147a7be-d6f6-458e-8b0d-70da7a3c7026-system-cni-dir\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392041 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-log-socket\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392051 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-run\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392068 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392071 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fd35650d-2c53-4230-8518-c4cb05b59558-dbus\") pod \"global-pull-secret-syncer-ldgnp\" (UID: \"fd35650d-2c53-4230-8518-c4cb05b59558\") " pod="kube-system/global-pull-secret-syncer-ldgnp" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392095 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-modprobe-d\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392097 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-log-socket\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392101 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392121 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-sysconfig\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392138 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-lib-modules\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392150 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-host\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392170 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99a7b227-aad8-414f-8668-70a5660d6a6e-host-slash\") pod \"iptables-alerter-bzvh4\" (UID: \"99a7b227-aad8-414f-8668-70a5660d6a6e\") " pod="openshift-network-operator/iptables-alerter-bzvh4" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392180 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-modprobe-d\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392216 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-run-ovn-kubernetes\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392224 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-host\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392243 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-var-lib-kubelet\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392265 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-lib-modules\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392243 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99a7b227-aad8-414f-8668-70a5660d6a6e-host-slash\") pod \"iptables-alerter-bzvh4\" (UID: \"99a7b227-aad8-414f-8668-70a5660d6a6e\") " pod="openshift-network-operator/iptables-alerter-bzvh4" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392296 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51fcfffe-f89d-4064-a0e3-45461cc62570-host-var-lib-kubelet\") pod \"multus-vdz2c\" (UID: \"51fcfffe-f89d-4064-a0e3-45461cc62570\") " pod="openshift-multus/multus-vdz2c" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392280 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1-etc-sysconfig\") pod \"tuned-m4s7g\" (UID: \"e98b6bfb-2f6b-4d7a-9541-7dfbb8a88fd1\") " pod="openshift-cluster-node-tuning-operator/tuned-m4s7g" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392267 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-host-run-ovn-kubernetes\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392333 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-run-openvswitch\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392269 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ea8559a-aa64-4d00-84c0-2c9405e32606-run-openvswitch\") pod \"ovnkube-node-9b8jf\" (UID: \"6ea8559a-aa64-4d00-84c0-2c9405e32606\") " pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392381 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c147a7be-d6f6-458e-8b0d-70da7a3c7026-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:42:27.393536 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:27.392430 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c147a7be-d6f6-458e-8b0d-70da7a3c7026-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rdtdb\" (UID: \"c147a7be-d6f6-458e-8b0d-70da7a3c7026\") " pod="openshift-multus/multus-additional-cni-plugins-rdtdb" Oct 29 22:42:28.404600 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:28.404571 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:28.966730 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:28.966700 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:28.968102 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:28.968080 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zb2ch" Oct 29 22:42:30.259348 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:30.259314 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:42:35.432830 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:35.432796 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:35.433192 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:35.433071 4821 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:42:39.392797 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:39.392766 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9b8jf" Oct 29 22:42:53.881688 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:42:53.881645 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:42:54.363248 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:42:54.363225 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jzjsw" Oct 29 22:43:00.262057 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:43:00.262021 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mbv7x" Oct 29 22:43:23.882348 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:43:23.882311 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:43:53.882361 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:43:53.882327 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:44:23.882924 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:44:23.882883 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:44:34.309022 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.308989 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd"] Oct 29 22:44:34.312317 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.312295 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd" Oct 29 22:44:34.344873 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.344850 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-dh6wr\"" Oct 29 22:44:34.360709 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.360691 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 29 22:44:34.361248 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.361234 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 29 22:44:34.367938 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.367919 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd"] Oct 29 22:44:34.373876 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.373853 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qgv5\" (UniqueName: \"kubernetes.io/projected/dd7dfc52-f51d-4764-bc34-b7ad01bca72e-kube-api-access-9qgv5\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd\" (UID: \"dd7dfc52-f51d-4764-bc34-b7ad01bca72e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd" Oct 29 22:44:34.373959 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.373892 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd7dfc52-f51d-4764-bc34-b7ad01bca72e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd\" (UID: \"dd7dfc52-f51d-4764-bc34-b7ad01bca72e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd" Oct 29 22:44:34.373959 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.373927 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd7dfc52-f51d-4764-bc34-b7ad01bca72e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd\" (UID: \"dd7dfc52-f51d-4764-bc34-b7ad01bca72e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd" Oct 29 22:44:34.427637 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.427614 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k"] Oct 29 22:44:34.430690 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.430675 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k" Oct 29 22:44:34.444646 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.444625 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-7qdvr\"" Oct 29 22:44:34.451636 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.451613 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k"] Oct 29 22:44:34.474466 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.474438 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qgv5\" (UniqueName: \"kubernetes.io/projected/dd7dfc52-f51d-4764-bc34-b7ad01bca72e-kube-api-access-9qgv5\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd\" (UID: \"dd7dfc52-f51d-4764-bc34-b7ad01bca72e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd" Oct 29 22:44:34.474636 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.474477 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49ce4f4e-0072-4a0c-b229-8766cde1bee0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k\" (UID: \"49ce4f4e-0072-4a0c-b229-8766cde1bee0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k" Oct 29 22:44:34.474636 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.474500 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd7dfc52-f51d-4764-bc34-b7ad01bca72e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd\" (UID: \"dd7dfc52-f51d-4764-bc34-b7ad01bca72e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd" Oct 29 22:44:34.474636 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.474534 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzfj9\" (UniqueName: \"kubernetes.io/projected/49ce4f4e-0072-4a0c-b229-8766cde1bee0-kube-api-access-jzfj9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k\" (UID: \"49ce4f4e-0072-4a0c-b229-8766cde1bee0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k" Oct 29 22:44:34.474636 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.474601 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd7dfc52-f51d-4764-bc34-b7ad01bca72e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd\" (UID: \"dd7dfc52-f51d-4764-bc34-b7ad01bca72e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd" Oct 29 22:44:34.474852 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.474712 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49ce4f4e-0072-4a0c-b229-8766cde1bee0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k\" (UID: \"49ce4f4e-0072-4a0c-b229-8766cde1bee0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k" Oct 29 22:44:34.474915 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.474848 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd7dfc52-f51d-4764-bc34-b7ad01bca72e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd\" (UID: \"dd7dfc52-f51d-4764-bc34-b7ad01bca72e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd" Oct 29 22:44:34.474969 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.474940 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd7dfc52-f51d-4764-bc34-b7ad01bca72e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd\" (UID: \"dd7dfc52-f51d-4764-bc34-b7ad01bca72e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd" Oct 29 22:44:34.497444 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.497418 4821 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 29 22:44:34.499729 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.499708 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qgv5\" (UniqueName: \"kubernetes.io/projected/dd7dfc52-f51d-4764-bc34-b7ad01bca72e-kube-api-access-9qgv5\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd\" (UID: \"dd7dfc52-f51d-4764-bc34-b7ad01bca72e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd" Oct 29 22:44:34.575418 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.575330 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49ce4f4e-0072-4a0c-b229-8766cde1bee0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k\" (UID: \"49ce4f4e-0072-4a0c-b229-8766cde1bee0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k" Oct 29 22:44:34.575418 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.575380 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzfj9\" (UniqueName: \"kubernetes.io/projected/49ce4f4e-0072-4a0c-b229-8766cde1bee0-kube-api-access-jzfj9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k\" (UID: \"49ce4f4e-0072-4a0c-b229-8766cde1bee0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k" Oct 29 22:44:34.575418 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.575410 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49ce4f4e-0072-4a0c-b229-8766cde1bee0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k\" (UID: \"49ce4f4e-0072-4a0c-b229-8766cde1bee0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k" Oct 29 22:44:34.575771 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.575750 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49ce4f4e-0072-4a0c-b229-8766cde1bee0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k\" (UID: \"49ce4f4e-0072-4a0c-b229-8766cde1bee0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k" Oct 29 22:44:34.575834 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.575787 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49ce4f4e-0072-4a0c-b229-8766cde1bee0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k\" (UID: \"49ce4f4e-0072-4a0c-b229-8766cde1bee0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k" Oct 29 22:44:34.599054 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.599025 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzfj9\" (UniqueName: \"kubernetes.io/projected/49ce4f4e-0072-4a0c-b229-8766cde1bee0-kube-api-access-jzfj9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k\" (UID: \"49ce4f4e-0072-4a0c-b229-8766cde1bee0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k" Oct 29 22:44:34.627145 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.627122 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd" Oct 29 22:44:34.745434 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.745379 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k" Oct 29 22:44:34.926073 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.926028 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd"] Oct 29 22:44:34.929421 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:44:34.929392 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7dfc52_f51d_4764_bc34_b7ad01bca72e.slice/crio-b46a616740872045ae7a0e25b56cd2f983f80a980928f81d82e2d9fbdd520907 WatchSource:0}: Error finding container b46a616740872045ae7a0e25b56cd2f983f80a980928f81d82e2d9fbdd520907: Status 404 returned error can't find the container with id b46a616740872045ae7a0e25b56cd2f983f80a980928f81d82e2d9fbdd520907 Oct 29 22:44:34.931398 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:34.931379 4821 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:44:35.040342 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:35.040300 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k"] Oct 29 22:44:35.043607 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:44:35.043580 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ce4f4e_0072_4a0c_b229_8766cde1bee0.slice/crio-e77169cdbf254c07c0b6db7c6a8c5ef195165ddb4b793d737fe6d3c8c6b0b491 WatchSource:0}: Error finding container e77169cdbf254c07c0b6db7c6a8c5ef195165ddb4b793d737fe6d3c8c6b0b491: Status 404 returned error can't find the container with id e77169cdbf254c07c0b6db7c6a8c5ef195165ddb4b793d737fe6d3c8c6b0b491 Oct 29 22:44:35.591674 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:35.591621 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k" event={"ID":"49ce4f4e-0072-4a0c-b229-8766cde1bee0","Type":"ContainerStarted","Data":"e77169cdbf254c07c0b6db7c6a8c5ef195165ddb4b793d737fe6d3c8c6b0b491"} Oct 29 22:44:35.593266 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:35.593236 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd" event={"ID":"dd7dfc52-f51d-4764-bc34-b7ad01bca72e","Type":"ContainerStarted","Data":"b46a616740872045ae7a0e25b56cd2f983f80a980928f81d82e2d9fbdd520907"} Oct 29 22:44:37.356144 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:37.356083 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq"] Oct 29 22:44:37.359602 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:37.359575 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq" Oct 29 22:44:37.386543 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:37.386346 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq"] Oct 29 22:44:37.386543 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:37.386507 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-rchxq\"" Oct 29 22:44:37.402714 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:37.402681 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khkx9\" (UniqueName: \"kubernetes.io/projected/5ad8739d-6452-40ba-b7cd-76b1452b3049-kube-api-access-khkx9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq\" (UID: \"5ad8739d-6452-40ba-b7cd-76b1452b3049\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq" Oct 29 22:44:37.402863 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:37.402730 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ad8739d-6452-40ba-b7cd-76b1452b3049-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq\" (UID: \"5ad8739d-6452-40ba-b7cd-76b1452b3049\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq" Oct 29 22:44:37.402863 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:37.402842 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ad8739d-6452-40ba-b7cd-76b1452b3049-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq\" (UID: \"5ad8739d-6452-40ba-b7cd-76b1452b3049\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq" Oct 29 22:44:37.503771 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:37.503634 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ad8739d-6452-40ba-b7cd-76b1452b3049-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq\" (UID: \"5ad8739d-6452-40ba-b7cd-76b1452b3049\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq" Oct 29 22:44:37.503771 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:37.503715 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ad8739d-6452-40ba-b7cd-76b1452b3049-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq\" (UID: \"5ad8739d-6452-40ba-b7cd-76b1452b3049\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq" Oct 29 22:44:37.503988 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:37.503793 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khkx9\" (UniqueName: \"kubernetes.io/projected/5ad8739d-6452-40ba-b7cd-76b1452b3049-kube-api-access-khkx9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq\" (UID: \"5ad8739d-6452-40ba-b7cd-76b1452b3049\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq" Oct 29 22:44:37.504149 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:37.504121 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ad8739d-6452-40ba-b7cd-76b1452b3049-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq\" (UID: \"5ad8739d-6452-40ba-b7cd-76b1452b3049\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq" Oct 29 22:44:37.504287 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:37.504220 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ad8739d-6452-40ba-b7cd-76b1452b3049-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq\" (UID: \"5ad8739d-6452-40ba-b7cd-76b1452b3049\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq" Oct 29 22:44:37.527800 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:37.527767 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khkx9\" (UniqueName: \"kubernetes.io/projected/5ad8739d-6452-40ba-b7cd-76b1452b3049-kube-api-access-khkx9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq\" (UID: \"5ad8739d-6452-40ba-b7cd-76b1452b3049\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq" Oct 29 22:44:37.678236 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:37.678174 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq" Oct 29 22:44:37.996319 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:37.996238 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq"] Oct 29 22:44:38.036208 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.036173 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg"] Oct 29 22:44:38.040859 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.040840 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg" Oct 29 22:44:38.055032 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.055008 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-kbnf6\"" Oct 29 22:44:38.059860 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.059829 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg"] Oct 29 22:44:38.108424 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.108387 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9d4v\" (UniqueName: \"kubernetes.io/projected/aa6e5fe5-65ec-431a-9690-ab6b19ec81a5-kube-api-access-p9d4v\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg\" (UID: \"aa6e5fe5-65ec-431a-9690-ab6b19ec81a5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg" Oct 29 22:44:38.108609 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.108464 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa6e5fe5-65ec-431a-9690-ab6b19ec81a5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg\" (UID: \"aa6e5fe5-65ec-431a-9690-ab6b19ec81a5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg" Oct 29 22:44:38.108609 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.108497 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa6e5fe5-65ec-431a-9690-ab6b19ec81a5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg\" (UID: \"aa6e5fe5-65ec-431a-9690-ab6b19ec81a5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg" Oct 29 22:44:38.135049 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.135006 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk"] Oct 29 22:44:38.138613 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.138586 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk" Oct 29 22:44:38.153385 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.153360 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-xwhs6\"" Oct 29 22:44:38.159067 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.159047 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk"] Oct 29 22:44:38.209453 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.209424 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68455002-0f90-4682-81ef-5280a8236347-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk\" (UID: \"68455002-0f90-4682-81ef-5280a8236347\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk" Oct 29 22:44:38.209619 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.209480 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68455002-0f90-4682-81ef-5280a8236347-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk\" (UID: \"68455002-0f90-4682-81ef-5280a8236347\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk" Oct 29 22:44:38.209619 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.209515 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa6e5fe5-65ec-431a-9690-ab6b19ec81a5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg\" (UID: \"aa6e5fe5-65ec-431a-9690-ab6b19ec81a5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg" Oct 29 22:44:38.209755 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.209611 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa6e5fe5-65ec-431a-9690-ab6b19ec81a5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg\" (UID: \"aa6e5fe5-65ec-431a-9690-ab6b19ec81a5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg" Oct 29 22:44:38.209755 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.209665 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbrj8\" (UniqueName: \"kubernetes.io/projected/68455002-0f90-4682-81ef-5280a8236347-kube-api-access-lbrj8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk\" (UID: \"68455002-0f90-4682-81ef-5280a8236347\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk" Oct 29 22:44:38.209755 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.209735 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9d4v\" (UniqueName: \"kubernetes.io/projected/aa6e5fe5-65ec-431a-9690-ab6b19ec81a5-kube-api-access-p9d4v\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg\" (UID: \"aa6e5fe5-65ec-431a-9690-ab6b19ec81a5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg" Oct 29 22:44:38.209915 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.209881 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa6e5fe5-65ec-431a-9690-ab6b19ec81a5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg\" (UID: \"aa6e5fe5-65ec-431a-9690-ab6b19ec81a5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg" Oct 29 22:44:38.210185 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.210158 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa6e5fe5-65ec-431a-9690-ab6b19ec81a5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg\" (UID: \"aa6e5fe5-65ec-431a-9690-ab6b19ec81a5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg" Oct 29 22:44:38.236801 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.236770 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9d4v\" (UniqueName: \"kubernetes.io/projected/aa6e5fe5-65ec-431a-9690-ab6b19ec81a5-kube-api-access-p9d4v\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg\" (UID: \"aa6e5fe5-65ec-431a-9690-ab6b19ec81a5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg" Oct 29 22:44:38.236973 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.236913 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s"] Oct 29 22:44:38.247228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.247141 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s" Oct 29 22:44:38.260007 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.259979 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s"] Oct 29 22:44:38.262192 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.262071 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-vj4fq\"" Oct 29 22:44:38.310415 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.310384 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04c85156-c8b2-418f-806b-b9ab7d596348-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s\" (UID: \"04c85156-c8b2-418f-806b-b9ab7d596348\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s" Oct 29 22:44:38.310415 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.310426 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7nwx\" (UniqueName: \"kubernetes.io/projected/04c85156-c8b2-418f-806b-b9ab7d596348-kube-api-access-q7nwx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s\" (UID: \"04c85156-c8b2-418f-806b-b9ab7d596348\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s" Oct 29 22:44:38.310655 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.310456 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04c85156-c8b2-418f-806b-b9ab7d596348-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s\" (UID: \"04c85156-c8b2-418f-806b-b9ab7d596348\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s" Oct 29 22:44:38.310655 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.310539 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbrj8\" (UniqueName: \"kubernetes.io/projected/68455002-0f90-4682-81ef-5280a8236347-kube-api-access-lbrj8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk\" (UID: \"68455002-0f90-4682-81ef-5280a8236347\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk" Oct 29 22:44:38.310767 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.310665 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68455002-0f90-4682-81ef-5280a8236347-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk\" (UID: \"68455002-0f90-4682-81ef-5280a8236347\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk" Oct 29 22:44:38.310767 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.310734 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68455002-0f90-4682-81ef-5280a8236347-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk\" (UID: \"68455002-0f90-4682-81ef-5280a8236347\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk" Oct 29 22:44:38.311090 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.311069 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68455002-0f90-4682-81ef-5280a8236347-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk\" (UID: \"68455002-0f90-4682-81ef-5280a8236347\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk" Oct 29 22:44:38.311378 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.311339 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68455002-0f90-4682-81ef-5280a8236347-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk\" (UID: \"68455002-0f90-4682-81ef-5280a8236347\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk" Oct 29 22:44:38.337177 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.337145 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbrj8\" (UniqueName: \"kubernetes.io/projected/68455002-0f90-4682-81ef-5280a8236347-kube-api-access-lbrj8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk\" (UID: \"68455002-0f90-4682-81ef-5280a8236347\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk" Oct 29 22:44:38.339015 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.338990 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx"] Oct 29 22:44:38.342713 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.342691 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx" Oct 29 22:44:38.362015 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.361983 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx"] Oct 29 22:44:38.362412 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.362053 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg" Oct 29 22:44:38.371427 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.371402 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-ddkpp\"" Oct 29 22:44:38.413278 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.412074 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b51154dd-a714-441a-9f27-254335c1910e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx\" (UID: \"b51154dd-a714-441a-9f27-254335c1910e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx" Oct 29 22:44:38.413278 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.412118 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lll9r\" (UniqueName: \"kubernetes.io/projected/b51154dd-a714-441a-9f27-254335c1910e-kube-api-access-lll9r\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx\" (UID: \"b51154dd-a714-441a-9f27-254335c1910e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx" Oct 29 22:44:38.413278 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.412149 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b51154dd-a714-441a-9f27-254335c1910e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx\" (UID: \"b51154dd-a714-441a-9f27-254335c1910e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx" Oct 29 22:44:38.413278 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.412194 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04c85156-c8b2-418f-806b-b9ab7d596348-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s\" (UID: \"04c85156-c8b2-418f-806b-b9ab7d596348\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s" Oct 29 22:44:38.413278 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.412247 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7nwx\" (UniqueName: \"kubernetes.io/projected/04c85156-c8b2-418f-806b-b9ab7d596348-kube-api-access-q7nwx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s\" (UID: \"04c85156-c8b2-418f-806b-b9ab7d596348\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s" Oct 29 22:44:38.413278 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.412277 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04c85156-c8b2-418f-806b-b9ab7d596348-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s\" (UID: \"04c85156-c8b2-418f-806b-b9ab7d596348\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s" Oct 29 22:44:38.413278 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.412700 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04c85156-c8b2-418f-806b-b9ab7d596348-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s\" (UID: \"04c85156-c8b2-418f-806b-b9ab7d596348\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s" Oct 29 22:44:38.413278 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.413006 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04c85156-c8b2-418f-806b-b9ab7d596348-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s\" (UID: \"04c85156-c8b2-418f-806b-b9ab7d596348\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s" Oct 29 22:44:38.442598 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.442568 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7nwx\" (UniqueName: \"kubernetes.io/projected/04c85156-c8b2-418f-806b-b9ab7d596348-kube-api-access-q7nwx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s\" (UID: \"04c85156-c8b2-418f-806b-b9ab7d596348\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s" Oct 29 22:44:38.458088 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.458061 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk" Oct 29 22:44:38.514500 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.513435 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b51154dd-a714-441a-9f27-254335c1910e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx\" (UID: \"b51154dd-a714-441a-9f27-254335c1910e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx" Oct 29 22:44:38.514500 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.513492 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lll9r\" (UniqueName: \"kubernetes.io/projected/b51154dd-a714-441a-9f27-254335c1910e-kube-api-access-lll9r\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx\" (UID: \"b51154dd-a714-441a-9f27-254335c1910e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx" Oct 29 22:44:38.514500 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.513523 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b51154dd-a714-441a-9f27-254335c1910e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx\" (UID: \"b51154dd-a714-441a-9f27-254335c1910e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx" Oct 29 22:44:38.514500 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.513969 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b51154dd-a714-441a-9f27-254335c1910e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx\" (UID: \"b51154dd-a714-441a-9f27-254335c1910e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx" Oct 29 22:44:38.515268 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.515223 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b51154dd-a714-441a-9f27-254335c1910e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx\" (UID: \"b51154dd-a714-441a-9f27-254335c1910e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx" Oct 29 22:44:38.541344 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.541315 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lll9r\" (UniqueName: \"kubernetes.io/projected/b51154dd-a714-441a-9f27-254335c1910e-kube-api-access-lll9r\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx\" (UID: \"b51154dd-a714-441a-9f27-254335c1910e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx" Oct 29 22:44:38.571218 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.570965 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s" Oct 29 22:44:38.607312 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.607278 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq" event={"ID":"5ad8739d-6452-40ba-b7cd-76b1452b3049","Type":"ContainerStarted","Data":"a8a9996d0a3bacae1fba763de3aa6ba6ea66aa0760ff7dd9098982e68019c44c"} Oct 29 22:44:38.663667 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.663639 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx" Oct 29 22:44:38.678732 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.678689 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg"] Oct 29 22:44:38.683117 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:44:38.683093 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa6e5fe5_65ec_431a_9690_ab6b19ec81a5.slice/crio-c881a1e056c556c0167b40e7db3b3ff74a978d4d303b1df3682afbe264b2b09a WatchSource:0}: Error finding container c881a1e056c556c0167b40e7db3b3ff74a978d4d303b1df3682afbe264b2b09a: Status 404 returned error can't find the container with id c881a1e056c556c0167b40e7db3b3ff74a978d4d303b1df3682afbe264b2b09a Oct 29 22:44:38.779405 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.779326 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk"] Oct 29 22:44:38.789532 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:44:38.789500 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68455002_0f90_4682_81ef_5280a8236347.slice/crio-0a88f197e53a1e1c34d0b8493d76cb40587e9bac429a4a5ab6a204c097abffd7 WatchSource:0}: Error finding container 0a88f197e53a1e1c34d0b8493d76cb40587e9bac429a4a5ab6a204c097abffd7: Status 404 returned error can't find the container with id 0a88f197e53a1e1c34d0b8493d76cb40587e9bac429a4a5ab6a204c097abffd7 Oct 29 22:44:38.885038 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.885008 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s"] Oct 29 22:44:38.889468 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:44:38.889434 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c85156_c8b2_418f_806b_b9ab7d596348.slice/crio-3960ae9622e64fc7ac16c3a129f29eb09a383cc0dcb9c6b8341140aff95bd8f3 WatchSource:0}: Error finding container 3960ae9622e64fc7ac16c3a129f29eb09a383cc0dcb9c6b8341140aff95bd8f3: Status 404 returned error can't find the container with id 3960ae9622e64fc7ac16c3a129f29eb09a383cc0dcb9c6b8341140aff95bd8f3 Oct 29 22:44:38.976738 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:38.976702 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx"] Oct 29 22:44:38.981960 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:44:38.981928 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb51154dd_a714_441a_9f27_254335c1910e.slice/crio-68b040955be49ff824a5127666ed54b4e0a25d7c9be922de1a04ba8d6ed1cf3a WatchSource:0}: Error finding container 68b040955be49ff824a5127666ed54b4e0a25d7c9be922de1a04ba8d6ed1cf3a: Status 404 returned error can't find the container with id 68b040955be49ff824a5127666ed54b4e0a25d7c9be922de1a04ba8d6ed1cf3a Oct 29 22:44:39.616236 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:39.616160 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx" event={"ID":"b51154dd-a714-441a-9f27-254335c1910e","Type":"ContainerStarted","Data":"68b040955be49ff824a5127666ed54b4e0a25d7c9be922de1a04ba8d6ed1cf3a"} Oct 29 22:44:39.618543 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:39.618515 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s" event={"ID":"04c85156-c8b2-418f-806b-b9ab7d596348","Type":"ContainerStarted","Data":"3960ae9622e64fc7ac16c3a129f29eb09a383cc0dcb9c6b8341140aff95bd8f3"} Oct 29 22:44:39.624068 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:39.624000 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg" event={"ID":"aa6e5fe5-65ec-431a-9690-ab6b19ec81a5","Type":"ContainerStarted","Data":"c881a1e056c556c0167b40e7db3b3ff74a978d4d303b1df3682afbe264b2b09a"} Oct 29 22:44:39.625500 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:39.625445 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk" event={"ID":"68455002-0f90-4682-81ef-5280a8236347","Type":"ContainerStarted","Data":"0a88f197e53a1e1c34d0b8493d76cb40587e9bac429a4a5ab6a204c097abffd7"} Oct 29 22:44:39.976731 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:39.976644 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr"] Oct 29 22:44:39.981985 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:39.981958 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.002558 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.001712 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2tp5f\"" Oct 29 22:44:40.002558 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.002502 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 29 22:44:40.013540 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.013488 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 29 22:44:40.015012 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.014933 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr"] Oct 29 22:44:40.026885 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.026856 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee04de80-4a76-4efd-8218-1f613b6130f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.027025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.026900 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfvtt\" (UniqueName: \"kubernetes.io/projected/ee04de80-4a76-4efd-8218-1f613b6130f0-kube-api-access-zfvtt\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.027025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.026929 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee04de80-4a76-4efd-8218-1f613b6130f0-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.027025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.026967 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee04de80-4a76-4efd-8218-1f613b6130f0-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.027025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.026999 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee04de80-4a76-4efd-8218-1f613b6130f0-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.027251 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.027043 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee04de80-4a76-4efd-8218-1f613b6130f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.027251 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.027076 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee04de80-4a76-4efd-8218-1f613b6130f0-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.027251 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.027105 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee04de80-4a76-4efd-8218-1f613b6130f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.027251 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.027135 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee04de80-4a76-4efd-8218-1f613b6130f0-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.097154 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.097096 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 29 22:44:40.104075 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.102919 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7"] Oct 29 22:44:40.107248 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.106984 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.126161 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.123844 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-7mm6v\"" Oct 29 22:44:40.128091 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.128063 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee04de80-4a76-4efd-8218-1f613b6130f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.128220 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.128113 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee04de80-4a76-4efd-8218-1f613b6130f0-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.128220 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.128181 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee04de80-4a76-4efd-8218-1f613b6130f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.128351 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.128242 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee04de80-4a76-4efd-8218-1f613b6130f0-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.128351 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.128280 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee04de80-4a76-4efd-8218-1f613b6130f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.128351 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.128321 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfvtt\" (UniqueName: \"kubernetes.io/projected/ee04de80-4a76-4efd-8218-1f613b6130f0-kube-api-access-zfvtt\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.128484 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.128359 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee04de80-4a76-4efd-8218-1f613b6130f0-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.128484 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.128420 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee04de80-4a76-4efd-8218-1f613b6130f0-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.128484 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.128469 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee04de80-4a76-4efd-8218-1f613b6130f0-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.128935 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.128906 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee04de80-4a76-4efd-8218-1f613b6130f0-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.129376 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.129354 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee04de80-4a76-4efd-8218-1f613b6130f0-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.129689 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.129668 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee04de80-4a76-4efd-8218-1f613b6130f0-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.129964 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.129943 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee04de80-4a76-4efd-8218-1f613b6130f0-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.130120 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.130094 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee04de80-4a76-4efd-8218-1f613b6130f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.145914 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.145865 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee04de80-4a76-4efd-8218-1f613b6130f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.148395 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.148372 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee04de80-4a76-4efd-8218-1f613b6130f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.155377 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.155310 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7"] Oct 29 22:44:40.168082 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.168057 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee04de80-4a76-4efd-8218-1f613b6130f0-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.171104 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.171037 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfvtt\" (UniqueName: \"kubernetes.io/projected/ee04de80-4a76-4efd-8218-1f613b6130f0-kube-api-access-zfvtt\") pod \"maas-default-gateway-openshift-default-6db586dd6c-lwrzr\" (UID: \"ee04de80-4a76-4efd-8218-1f613b6130f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.229276 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.229179 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2985a058-86e6-4d98-8bc5-307e78b3967c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.229446 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.229298 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89x7d\" (UniqueName: \"kubernetes.io/projected/2985a058-86e6-4d98-8bc5-307e78b3967c-kube-api-access-89x7d\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.229446 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.229331 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2985a058-86e6-4d98-8bc5-307e78b3967c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.229559 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.229453 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2985a058-86e6-4d98-8bc5-307e78b3967c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.229559 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.229484 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2985a058-86e6-4d98-8bc5-307e78b3967c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.229651 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.229573 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2985a058-86e6-4d98-8bc5-307e78b3967c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.229651 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.229599 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2985a058-86e6-4d98-8bc5-307e78b3967c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.229651 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.229626 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2985a058-86e6-4d98-8bc5-307e78b3967c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.229651 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.229648 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2985a058-86e6-4d98-8bc5-307e78b3967c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.273103 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.271856 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6"] Oct 29 22:44:40.281022 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.280988 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6" Oct 29 22:44:40.298675 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.298644 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6"] Oct 29 22:44:40.311497 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.311470 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:40.333791 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.330662 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2985a058-86e6-4d98-8bc5-307e78b3967c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.333791 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.330710 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2985a058-86e6-4d98-8bc5-307e78b3967c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.333791 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.330749 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2985a058-86e6-4d98-8bc5-307e78b3967c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.333791 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.330779 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/379859a0-ae5f-4423-9157-13272247ed50-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6\" (UID: \"379859a0-ae5f-4423-9157-13272247ed50\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6" Oct 29 22:44:40.333791 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.330842 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm4jr\" (UniqueName: \"kubernetes.io/projected/379859a0-ae5f-4423-9157-13272247ed50-kube-api-access-jm4jr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6\" (UID: \"379859a0-ae5f-4423-9157-13272247ed50\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6" Oct 29 22:44:40.333791 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.330874 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2985a058-86e6-4d98-8bc5-307e78b3967c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.333791 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.330910 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89x7d\" (UniqueName: \"kubernetes.io/projected/2985a058-86e6-4d98-8bc5-307e78b3967c-kube-api-access-89x7d\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.333791 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.330940 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2985a058-86e6-4d98-8bc5-307e78b3967c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.333791 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.330965 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2985a058-86e6-4d98-8bc5-307e78b3967c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.333791 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.330991 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2985a058-86e6-4d98-8bc5-307e78b3967c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.333791 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.331035 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2985a058-86e6-4d98-8bc5-307e78b3967c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.333791 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.331070 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/379859a0-ae5f-4423-9157-13272247ed50-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6\" (UID: \"379859a0-ae5f-4423-9157-13272247ed50\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6" Oct 29 22:44:40.333791 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.331468 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2985a058-86e6-4d98-8bc5-307e78b3967c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.333791 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.331718 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2985a058-86e6-4d98-8bc5-307e78b3967c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.333791 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.332669 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2985a058-86e6-4d98-8bc5-307e78b3967c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.333791 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.333553 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2985a058-86e6-4d98-8bc5-307e78b3967c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.334812 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.334699 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2985a058-86e6-4d98-8bc5-307e78b3967c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.341234 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.341142 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2985a058-86e6-4d98-8bc5-307e78b3967c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.342785 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.342706 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2985a058-86e6-4d98-8bc5-307e78b3967c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.358152 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.358100 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2985a058-86e6-4d98-8bc5-307e78b3967c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.359785 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.359765 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89x7d\" (UniqueName: \"kubernetes.io/projected/2985a058-86e6-4d98-8bc5-307e78b3967c-kube-api-access-89x7d\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7\" (UID: \"2985a058-86e6-4d98-8bc5-307e78b3967c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.412566 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.410513 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn"] Oct 29 22:44:40.418172 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.417571 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn" Oct 29 22:44:40.430631 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.430420 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:40.432670 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.431536 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/379859a0-ae5f-4423-9157-13272247ed50-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6\" (UID: \"379859a0-ae5f-4423-9157-13272247ed50\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6" Oct 29 22:44:40.432670 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.431596 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm4jr\" (UniqueName: \"kubernetes.io/projected/379859a0-ae5f-4423-9157-13272247ed50-kube-api-access-jm4jr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6\" (UID: \"379859a0-ae5f-4423-9157-13272247ed50\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6" Oct 29 22:44:40.432670 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.431659 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/379859a0-ae5f-4423-9157-13272247ed50-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6\" (UID: \"379859a0-ae5f-4423-9157-13272247ed50\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6" Oct 29 22:44:40.432670 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.432024 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/379859a0-ae5f-4423-9157-13272247ed50-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6\" (UID: \"379859a0-ae5f-4423-9157-13272247ed50\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6" Oct 29 22:44:40.432670 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.432286 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/379859a0-ae5f-4423-9157-13272247ed50-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6\" (UID: \"379859a0-ae5f-4423-9157-13272247ed50\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6" Oct 29 22:44:40.436895 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.436775 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn"] Oct 29 22:44:40.459785 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.459724 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm4jr\" (UniqueName: \"kubernetes.io/projected/379859a0-ae5f-4423-9157-13272247ed50-kube-api-access-jm4jr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6\" (UID: \"379859a0-ae5f-4423-9157-13272247ed50\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6" Oct 29 22:44:40.532598 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.532406 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a3e1c21-7250-4d3e-8025-e444bd340e00-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn\" (UID: \"8a3e1c21-7250-4d3e-8025-e444bd340e00\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn" Oct 29 22:44:40.532598 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.532466 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mc5n\" (UniqueName: \"kubernetes.io/projected/8a3e1c21-7250-4d3e-8025-e444bd340e00-kube-api-access-2mc5n\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn\" (UID: \"8a3e1c21-7250-4d3e-8025-e444bd340e00\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn" Oct 29 22:44:40.532598 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.532498 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a3e1c21-7250-4d3e-8025-e444bd340e00-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn\" (UID: \"8a3e1c21-7250-4d3e-8025-e444bd340e00\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn" Oct 29 22:44:40.614584 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.614341 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6" Oct 29 22:44:40.635250 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.633637 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mc5n\" (UniqueName: \"kubernetes.io/projected/8a3e1c21-7250-4d3e-8025-e444bd340e00-kube-api-access-2mc5n\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn\" (UID: \"8a3e1c21-7250-4d3e-8025-e444bd340e00\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn" Oct 29 22:44:40.635250 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.633693 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a3e1c21-7250-4d3e-8025-e444bd340e00-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn\" (UID: \"8a3e1c21-7250-4d3e-8025-e444bd340e00\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn" Oct 29 22:44:40.635250 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.633747 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a3e1c21-7250-4d3e-8025-e444bd340e00-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn\" (UID: \"8a3e1c21-7250-4d3e-8025-e444bd340e00\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn" Oct 29 22:44:40.635250 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.634105 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a3e1c21-7250-4d3e-8025-e444bd340e00-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn\" (UID: \"8a3e1c21-7250-4d3e-8025-e444bd340e00\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn" Oct 29 22:44:40.635866 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.635312 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a3e1c21-7250-4d3e-8025-e444bd340e00-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn\" (UID: \"8a3e1c21-7250-4d3e-8025-e444bd340e00\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn" Oct 29 22:44:40.662124 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.662077 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mc5n\" (UniqueName: \"kubernetes.io/projected/8a3e1c21-7250-4d3e-8025-e444bd340e00-kube-api-access-2mc5n\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn\" (UID: \"8a3e1c21-7250-4d3e-8025-e444bd340e00\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn" Oct 29 22:44:40.665746 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.665601 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr"] Oct 29 22:44:40.670855 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:44:40.670821 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee04de80_4a76_4efd_8218_1f613b6130f0.slice/crio-8d36bcb6810ceeb8b9494e4436050a8a71dedb3497560ee426f795f7334e71db WatchSource:0}: Error finding container 8d36bcb6810ceeb8b9494e4436050a8a71dedb3497560ee426f795f7334e71db: Status 404 returned error can't find the container with id 8d36bcb6810ceeb8b9494e4436050a8a71dedb3497560ee426f795f7334e71db Oct 29 22:44:40.746680 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.746607 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7"] Oct 29 22:44:40.746830 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.746695 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn" Oct 29 22:44:40.754812 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:44:40.754782 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2985a058_86e6_4d98_8bc5_307e78b3967c.slice/crio-b947220c6222efe046fd8a5e0611f375dadfea2e17ad5d1a56fb10263b243641 WatchSource:0}: Error finding container b947220c6222efe046fd8a5e0611f375dadfea2e17ad5d1a56fb10263b243641: Status 404 returned error can't find the container with id b947220c6222efe046fd8a5e0611f375dadfea2e17ad5d1a56fb10263b243641 Oct 29 22:44:40.953627 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:40.953584 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6"] Oct 29 22:44:41.075119 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:41.075047 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn"] Oct 29 22:44:41.081319 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:44:41.081284 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3e1c21_7250_4d3e_8025_e444bd340e00.slice/crio-4fb3f72f99f2540fccc1f8099116746afdff1b8e1ba8c6a0fc881825608c33b2 WatchSource:0}: Error finding container 4fb3f72f99f2540fccc1f8099116746afdff1b8e1ba8c6a0fc881825608c33b2: Status 404 returned error can't find the container with id 4fb3f72f99f2540fccc1f8099116746afdff1b8e1ba8c6a0fc881825608c33b2 Oct 29 22:44:41.673174 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:41.673137 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" event={"ID":"ee04de80-4a76-4efd-8218-1f613b6130f0","Type":"ContainerStarted","Data":"8d36bcb6810ceeb8b9494e4436050a8a71dedb3497560ee426f795f7334e71db"} Oct 29 22:44:41.683447 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:41.683407 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" event={"ID":"2985a058-86e6-4d98-8bc5-307e78b3967c","Type":"ContainerStarted","Data":"b947220c6222efe046fd8a5e0611f375dadfea2e17ad5d1a56fb10263b243641"} Oct 29 22:44:41.695338 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:41.695292 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6" event={"ID":"379859a0-ae5f-4423-9157-13272247ed50","Type":"ContainerStarted","Data":"17e8c7e8d10a6b959809fcb77ea2127defbdbef5b4a4ceaeacc39e093d7e764e"} Oct 29 22:44:41.714233 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:41.714170 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn" event={"ID":"8a3e1c21-7250-4d3e-8025-e444bd340e00","Type":"ContainerStarted","Data":"4fb3f72f99f2540fccc1f8099116746afdff1b8e1ba8c6a0fc881825608c33b2"} Oct 29 22:44:49.810700 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:49.810664 4821 generic.go:356] "Generic (PLEG): container finished" podID="dd7dfc52-f51d-4764-bc34-b7ad01bca72e" containerID="b1c1af99aad19061d18c8df92c3449fa1842ad59af561b4a70367bea1e00ad5f" exitCode=0 Oct 29 22:44:49.811332 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:49.810753 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd" event={"ID":"dd7dfc52-f51d-4764-bc34-b7ad01bca72e","Type":"ContainerDied","Data":"b1c1af99aad19061d18c8df92c3449fa1842ad59af561b4a70367bea1e00ad5f"} Oct 29 22:44:49.820112 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:49.819993 4821 generic.go:356] "Generic (PLEG): container finished" podID="5ad8739d-6452-40ba-b7cd-76b1452b3049" containerID="6ea0a14ac4b2cd6395d96153d0c7a8e8b2677597c4f3d599e081b673a1c128b4" exitCode=0 Oct 29 22:44:49.820112 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:49.820035 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq" event={"ID":"5ad8739d-6452-40ba-b7cd-76b1452b3049","Type":"ContainerDied","Data":"6ea0a14ac4b2cd6395d96153d0c7a8e8b2677597c4f3d599e081b673a1c128b4"} Oct 29 22:44:50.826676 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:50.826606 4821 generic.go:356] "Generic (PLEG): container finished" podID="68455002-0f90-4682-81ef-5280a8236347" containerID="f6e1abb579ba766191a6e5c1c345cb8f67c0875041f9b2747039851dfdb77d61" exitCode=0 Oct 29 22:44:50.827167 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:50.826701 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk" event={"ID":"68455002-0f90-4682-81ef-5280a8236347","Type":"ContainerDied","Data":"f6e1abb579ba766191a6e5c1c345cb8f67c0875041f9b2747039851dfdb77d61"} Oct 29 22:44:50.828953 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:50.828927 4821 generic.go:356] "Generic (PLEG): container finished" podID="49ce4f4e-0072-4a0c-b229-8766cde1bee0" containerID="b43a03d63f75eefb334eb320cf50fd8220e60c2aa4476212f90d80dadf6168bf" exitCode=0 Oct 29 22:44:50.829059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:50.829029 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k" event={"ID":"49ce4f4e-0072-4a0c-b229-8766cde1bee0","Type":"ContainerDied","Data":"b43a03d63f75eefb334eb320cf50fd8220e60c2aa4476212f90d80dadf6168bf"} Oct 29 22:44:50.830869 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:50.830848 4821 generic.go:356] "Generic (PLEG): container finished" podID="aa6e5fe5-65ec-431a-9690-ab6b19ec81a5" containerID="4e2b7fd91caf29dd365e89163c79c2772e8b1387e35e42a81766125519e761b5" exitCode=0 Oct 29 22:44:50.830959 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:50.830880 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg" event={"ID":"aa6e5fe5-65ec-431a-9690-ab6b19ec81a5","Type":"ContainerDied","Data":"4e2b7fd91caf29dd365e89163c79c2772e8b1387e35e42a81766125519e761b5"} Oct 29 22:44:51.353903 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:51.353868 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:44:51.354016 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:51.353957 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:44:51.354016 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:51.354002 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:44:51.366679 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:51.366646 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:44:51.366769 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:51.366727 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:44:51.366807 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:51.366772 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:44:51.836951 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:51.836911 4821 generic.go:356] "Generic (PLEG): container finished" podID="68455002-0f90-4682-81ef-5280a8236347" containerID="bfb8b986a3e2313c57f4aba57a5d983524fbc0878a4240d139257b14837aa1fe" exitCode=0 Oct 29 22:44:51.837411 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:51.836998 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk" event={"ID":"68455002-0f90-4682-81ef-5280a8236347","Type":"ContainerDied","Data":"bfb8b986a3e2313c57f4aba57a5d983524fbc0878a4240d139257b14837aa1fe"} Oct 29 22:44:51.838785 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:51.838757 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" event={"ID":"ee04de80-4a76-4efd-8218-1f613b6130f0","Type":"ContainerStarted","Data":"5b3db4492028207bd923310fa379c656ebdcee2d88250f136d3b8fbfbf90f6eb"} Oct 29 22:44:51.840505 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:51.840478 4821 generic.go:356] "Generic (PLEG): container finished" podID="b51154dd-a714-441a-9f27-254335c1910e" containerID="22fe632609f0f4aedde08661c85507f3f9b57f6faeb3d51e197c0b2c84c3e9ef" exitCode=0 Oct 29 22:44:51.840610 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:51.840571 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx" event={"ID":"b51154dd-a714-441a-9f27-254335c1910e","Type":"ContainerDied","Data":"22fe632609f0f4aedde08661c85507f3f9b57f6faeb3d51e197c0b2c84c3e9ef"} Oct 29 22:44:51.843079 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:51.843048 4821 generic.go:356] "Generic (PLEG): container finished" podID="04c85156-c8b2-418f-806b-b9ab7d596348" containerID="428f9baa898b3befc2d70e57716e594d7da41768f0484a4beae777a65e581e89" exitCode=0 Oct 29 22:44:51.843165 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:51.843140 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s" event={"ID":"04c85156-c8b2-418f-806b-b9ab7d596348","Type":"ContainerDied","Data":"428f9baa898b3befc2d70e57716e594d7da41768f0484a4beae777a65e581e89"} Oct 29 22:44:51.845913 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:51.845887 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" event={"ID":"2985a058-86e6-4d98-8bc5-307e78b3967c","Type":"ContainerStarted","Data":"4b89a8cec5af5a79cad3a42216abe4eff8a2c064670e108daf251ee12c7d764c"} Oct 29 22:44:51.847686 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:51.847661 4821 generic.go:356] "Generic (PLEG): container finished" podID="379859a0-ae5f-4423-9157-13272247ed50" containerID="555f7b3ee5bd376cbc8f2f096940e6347c13f8868185df874bbc945f12be8068" exitCode=0 Oct 29 22:44:51.847788 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:51.847764 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6" event={"ID":"379859a0-ae5f-4423-9157-13272247ed50","Type":"ContainerDied","Data":"555f7b3ee5bd376cbc8f2f096940e6347c13f8868185df874bbc945f12be8068"} Oct 29 22:44:51.849631 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:51.849577 4821 generic.go:356] "Generic (PLEG): container finished" podID="8a3e1c21-7250-4d3e-8025-e444bd340e00" containerID="f0b2b1137b918d68ed3eace39b5860c737907fc62117e908ed635602a947ef59" exitCode=0 Oct 29 22:44:51.849703 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:51.849652 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn" event={"ID":"8a3e1c21-7250-4d3e-8025-e444bd340e00","Type":"ContainerDied","Data":"f0b2b1137b918d68ed3eace39b5860c737907fc62117e908ed635602a947ef59"} Oct 29 22:44:51.853694 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:51.853663 4821 generic.go:356] "Generic (PLEG): container finished" podID="5ad8739d-6452-40ba-b7cd-76b1452b3049" containerID="abd98d5ee98b128ed786eaba73153b7bdabb9011149db00ceea88479c74a8b50" exitCode=0 Oct 29 22:44:51.853800 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:51.853703 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq" event={"ID":"5ad8739d-6452-40ba-b7cd-76b1452b3049","Type":"ContainerDied","Data":"abd98d5ee98b128ed786eaba73153b7bdabb9011149db00ceea88479c74a8b50"} Oct 29 22:44:52.094729 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:52.094587 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" podStartSLOduration=2.4147435059999998 podStartE2EDuration="13.094567021s" totalImagesPullingTime="10.679823515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:52.093394941 +0000 UTC m=+146.033942851" watchObservedRunningTime="2026-10-29 22:44:52.094567021 +0000 UTC m=+146.035114933" Oct 29 22:44:52.137558 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:52.137496 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" podStartSLOduration=1.5338370650000002 podStartE2EDuration="12.137476678s" totalImagesPullingTime="10.603639613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:52.13711051 +0000 UTC m=+146.077658420" watchObservedRunningTime="2026-10-29 22:44:52.137476678 +0000 UTC m=+146.078024590" Oct 29 22:44:52.312778 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:52.312690 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:52.319991 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:52.319966 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:52.432650 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:52.432255 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:52.438809 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:52.438785 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:52.882849 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:52.882813 4821 generic.go:356] "Generic (PLEG): container finished" podID="aa6e5fe5-65ec-431a-9690-ab6b19ec81a5" containerID="e5f91103ca30e3cdc64e53b2d80b3169c4e9378f4b72e773b1fea7edcca90d8a" exitCode=0 Oct 29 22:44:52.883304 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:52.882953 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg" event={"ID":"aa6e5fe5-65ec-431a-9690-ab6b19ec81a5","Type":"ContainerDied","Data":"e5f91103ca30e3cdc64e53b2d80b3169c4e9378f4b72e773b1fea7edcca90d8a"} Oct 29 22:44:52.895882 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:52.895341 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx" event={"ID":"b51154dd-a714-441a-9f27-254335c1910e","Type":"ContainerStarted","Data":"0d39874e2c30d31039be1cc9f0a7df10468a4b7a5cc3db4d2ed8aced4f531670"} Oct 29 22:44:52.904614 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:52.904185 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:52.904614 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:52.904579 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:52.905010 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:52.904993 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:44:52.905532 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:52.905513 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7" Oct 29 22:44:53.903578 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:44:53.902828 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:44:53.921674 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:53.921473 4821 generic.go:356] "Generic (PLEG): container finished" podID="b51154dd-a714-441a-9f27-254335c1910e" containerID="0d39874e2c30d31039be1cc9f0a7df10468a4b7a5cc3db4d2ed8aced4f531670" exitCode=0 Oct 29 22:44:53.921674 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:53.921562 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx" event={"ID":"b51154dd-a714-441a-9f27-254335c1910e","Type":"ContainerDied","Data":"0d39874e2c30d31039be1cc9f0a7df10468a4b7a5cc3db4d2ed8aced4f531670"} Oct 29 22:44:53.924856 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:53.924826 4821 generic.go:356] "Generic (PLEG): container finished" podID="04c85156-c8b2-418f-806b-b9ab7d596348" containerID="5435897906e56ec46baab72f044b1b05596e8dbf7e7c0575704b55c6ea96a086" exitCode=0 Oct 29 22:44:53.925946 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:53.925320 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s" event={"ID":"04c85156-c8b2-418f-806b-b9ab7d596348","Type":"ContainerDied","Data":"5435897906e56ec46baab72f044b1b05596e8dbf7e7c0575704b55c6ea96a086"} Oct 29 22:44:54.933462 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:54.933421 4821 generic.go:356] "Generic (PLEG): container finished" podID="dd7dfc52-f51d-4764-bc34-b7ad01bca72e" containerID="8dec91eefc66192180a25430e55f390546c55135f933a3afa51ed2015756a373" exitCode=0 Oct 29 22:44:54.934302 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:54.934277 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd" event={"ID":"dd7dfc52-f51d-4764-bc34-b7ad01bca72e","Type":"ContainerDied","Data":"8dec91eefc66192180a25430e55f390546c55135f933a3afa51ed2015756a373"} Oct 29 22:44:55.948170 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:55.948136 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k" event={"ID":"49ce4f4e-0072-4a0c-b229-8766cde1bee0","Type":"ContainerStarted","Data":"dc13b22aa138212f6c1583e42777b9c7a19163a2fbea53925122ea34cf551d1d"} Oct 29 22:44:56.964566 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:56.963562 4821 generic.go:356] "Generic (PLEG): container finished" podID="49ce4f4e-0072-4a0c-b229-8766cde1bee0" containerID="dc13b22aa138212f6c1583e42777b9c7a19163a2fbea53925122ea34cf551d1d" exitCode=0 Oct 29 22:44:56.964566 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:56.963672 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k" event={"ID":"49ce4f4e-0072-4a0c-b229-8766cde1bee0","Type":"ContainerDied","Data":"dc13b22aa138212f6c1583e42777b9c7a19163a2fbea53925122ea34cf551d1d"} Oct 29 22:44:57.993359 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:57.993321 4821 generic.go:356] "Generic (PLEG): container finished" podID="379859a0-ae5f-4423-9157-13272247ed50" containerID="ae2bb51b1e88a412fea416fd49aab361deb4a0308b66a10c32885a827fc303c2" exitCode=0 Oct 29 22:44:57.993819 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:57.993401 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6" event={"ID":"379859a0-ae5f-4423-9157-13272247ed50","Type":"ContainerDied","Data":"ae2bb51b1e88a412fea416fd49aab361deb4a0308b66a10c32885a827fc303c2"} Oct 29 22:44:59.010308 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:59.010268 4821 generic.go:356] "Generic (PLEG): container finished" podID="8a3e1c21-7250-4d3e-8025-e444bd340e00" containerID="70762da392c614e0bec92552a576285ef9bc9a1921024496d0ca408a0712fe25" exitCode=0 Oct 29 22:44:59.010799 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:44:59.010385 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn" event={"ID":"8a3e1c21-7250-4d3e-8025-e444bd340e00","Type":"ContainerDied","Data":"70762da392c614e0bec92552a576285ef9bc9a1921024496d0ca408a0712fe25"} Oct 29 22:45:11.125132 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:11.125098 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn" event={"ID":"8a3e1c21-7250-4d3e-8025-e444bd340e00","Type":"ContainerStarted","Data":"e0643df6fe0239a87e0e4166c4baa10bfbeb6d6353ccdddcc73611eee430fa4a"} Oct 29 22:45:11.129087 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:11.129055 4821 generic.go:356] "Generic (PLEG): container finished" podID="5ad8739d-6452-40ba-b7cd-76b1452b3049" containerID="6de60b0a56e441a2e5c19a74147dc007ca3539fd33335ea22ee1dc51c3cf9078" exitCode=0 Oct 29 22:45:11.129371 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:11.129338 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq" event={"ID":"5ad8739d-6452-40ba-b7cd-76b1452b3049","Type":"ContainerDied","Data":"6de60b0a56e441a2e5c19a74147dc007ca3539fd33335ea22ee1dc51c3cf9078"} Oct 29 22:45:11.133063 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:11.133035 4821 generic.go:356] "Generic (PLEG): container finished" podID="aa6e5fe5-65ec-431a-9690-ab6b19ec81a5" containerID="06af743e245ddf878c7f88b83d511130d9d9f83c0b49761b203117d5bef563e4" exitCode=0 Oct 29 22:45:11.133166 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:11.133123 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg" event={"ID":"aa6e5fe5-65ec-431a-9690-ab6b19ec81a5","Type":"ContainerDied","Data":"06af743e245ddf878c7f88b83d511130d9d9f83c0b49761b203117d5bef563e4"} Oct 29 22:45:11.136509 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:11.136185 4821 generic.go:356] "Generic (PLEG): container finished" podID="68455002-0f90-4682-81ef-5280a8236347" containerID="34cf86a2cee64d441d489454f85155a4b357ab19d1c7aaeb06300027600af791" exitCode=0 Oct 29 22:45:11.136509 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:11.136325 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk" event={"ID":"68455002-0f90-4682-81ef-5280a8236347","Type":"ContainerDied","Data":"34cf86a2cee64d441d489454f85155a4b357ab19d1c7aaeb06300027600af791"} Oct 29 22:45:11.169016 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:11.168948 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn" podStartSLOduration=3.214758981 podStartE2EDuration="31.168931833s" totalImagesPullingTime="27.954172852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:11.168369457 +0000 UTC m=+165.108917368" watchObservedRunningTime="2026-10-29 22:45:11.168931833 +0000 UTC m=+165.109479745" Oct 29 22:45:12.147303 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.147119 4821 generic.go:356] "Generic (PLEG): container finished" podID="b51154dd-a714-441a-9f27-254335c1910e" containerID="45d010ab92510beb57ce9043aa4ced8fbfd0f2b41b22c21f399c3a2491152e7c" exitCode=0 Oct 29 22:45:12.147303 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.147183 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx" event={"ID":"b51154dd-a714-441a-9f27-254335c1910e","Type":"ContainerDied","Data":"45d010ab92510beb57ce9043aa4ced8fbfd0f2b41b22c21f399c3a2491152e7c"} Oct 29 22:45:12.150345 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.150317 4821 generic.go:356] "Generic (PLEG): container finished" podID="04c85156-c8b2-418f-806b-b9ab7d596348" containerID="cce2e257c0c6584580bb0e5c26408089a4d30249c2e4e1f4f4d3da92ddad4204" exitCode=0 Oct 29 22:45:12.150500 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.150424 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s" event={"ID":"04c85156-c8b2-418f-806b-b9ab7d596348","Type":"ContainerDied","Data":"cce2e257c0c6584580bb0e5c26408089a4d30249c2e4e1f4f4d3da92ddad4204"} Oct 29 22:45:12.153440 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.153414 4821 generic.go:356] "Generic (PLEG): container finished" podID="8a3e1c21-7250-4d3e-8025-e444bd340e00" containerID="e0643df6fe0239a87e0e4166c4baa10bfbeb6d6353ccdddcc73611eee430fa4a" exitCode=0 Oct 29 22:45:12.153838 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.153818 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn" event={"ID":"8a3e1c21-7250-4d3e-8025-e444bd340e00","Type":"ContainerDied","Data":"e0643df6fe0239a87e0e4166c4baa10bfbeb6d6353ccdddcc73611eee430fa4a"} Oct 29 22:45:12.343937 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.343910 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk" Oct 29 22:45:12.364251 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.364222 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq" Oct 29 22:45:12.376385 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.376339 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg" Oct 29 22:45:12.477678 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.477651 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa6e5fe5-65ec-431a-9690-ab6b19ec81a5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa6e5fe5-65ec-431a-9690-ab6b19ec81a5-bundle\") pod \"aa6e5fe5-65ec-431a-9690-ab6b19ec81a5\" (UID: \"aa6e5fe5-65ec-431a-9690-ab6b19ec81a5\") " Oct 29 22:45:12.477858 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.477711 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa6e5fe5-65ec-431a-9690-ab6b19ec81a5-kube-api-access-p9d4v\" (UniqueName: \"kubernetes.io/projected/aa6e5fe5-65ec-431a-9690-ab6b19ec81a5-kube-api-access-p9d4v\") pod \"aa6e5fe5-65ec-431a-9690-ab6b19ec81a5\" (UID: \"aa6e5fe5-65ec-431a-9690-ab6b19ec81a5\") " Oct 29 22:45:12.477858 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.477756 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68455002-0f90-4682-81ef-5280a8236347-bundle\" (UniqueName: \"kubernetes.io/empty-dir/68455002-0f90-4682-81ef-5280a8236347-bundle\") pod \"68455002-0f90-4682-81ef-5280a8236347\" (UID: \"68455002-0f90-4682-81ef-5280a8236347\") " Oct 29 22:45:12.477858 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.477784 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad8739d-6452-40ba-b7cd-76b1452b3049-kube-api-access-khkx9\" (UniqueName: \"kubernetes.io/projected/5ad8739d-6452-40ba-b7cd-76b1452b3049-kube-api-access-khkx9\") pod \"5ad8739d-6452-40ba-b7cd-76b1452b3049\" (UID: \"5ad8739d-6452-40ba-b7cd-76b1452b3049\") " Oct 29 22:45:12.477858 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.477814 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68455002-0f90-4682-81ef-5280a8236347-util\" (UniqueName: \"kubernetes.io/empty-dir/68455002-0f90-4682-81ef-5280a8236347-util\") pod \"68455002-0f90-4682-81ef-5280a8236347\" (UID: \"68455002-0f90-4682-81ef-5280a8236347\") " Oct 29 22:45:12.477858 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.477850 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68455002-0f90-4682-81ef-5280a8236347-kube-api-access-lbrj8\" (UniqueName: \"kubernetes.io/projected/68455002-0f90-4682-81ef-5280a8236347-kube-api-access-lbrj8\") pod \"68455002-0f90-4682-81ef-5280a8236347\" (UID: \"68455002-0f90-4682-81ef-5280a8236347\") " Oct 29 22:45:12.478128 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.477890 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ad8739d-6452-40ba-b7cd-76b1452b3049-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ad8739d-6452-40ba-b7cd-76b1452b3049-bundle\") pod \"5ad8739d-6452-40ba-b7cd-76b1452b3049\" (UID: \"5ad8739d-6452-40ba-b7cd-76b1452b3049\") " Oct 29 22:45:12.478128 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.477939 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa6e5fe5-65ec-431a-9690-ab6b19ec81a5-util\" (UniqueName: \"kubernetes.io/empty-dir/aa6e5fe5-65ec-431a-9690-ab6b19ec81a5-util\") pod \"aa6e5fe5-65ec-431a-9690-ab6b19ec81a5\" (UID: \"aa6e5fe5-65ec-431a-9690-ab6b19ec81a5\") " Oct 29 22:45:12.478128 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.477983 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ad8739d-6452-40ba-b7cd-76b1452b3049-util\" (UniqueName: \"kubernetes.io/empty-dir/5ad8739d-6452-40ba-b7cd-76b1452b3049-util\") pod \"5ad8739d-6452-40ba-b7cd-76b1452b3049\" (UID: \"5ad8739d-6452-40ba-b7cd-76b1452b3049\") " Oct 29 22:45:12.478299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.478143 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6e5fe5-65ec-431a-9690-ab6b19ec81a5-bundle" pod "aa6e5fe5-65ec-431a-9690-ab6b19ec81a5" (UID: "aa6e5fe5-65ec-431a-9690-ab6b19ec81a5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:12.478857 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.478443 4821 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa6e5fe5-65ec-431a-9690-ab6b19ec81a5-bundle\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:12.478857 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.478795 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68455002-0f90-4682-81ef-5280a8236347-bundle" pod "68455002-0f90-4682-81ef-5280a8236347" (UID: "68455002-0f90-4682-81ef-5280a8236347"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:12.479372 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.479338 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad8739d-6452-40ba-b7cd-76b1452b3049-bundle" pod "5ad8739d-6452-40ba-b7cd-76b1452b3049" (UID: "5ad8739d-6452-40ba-b7cd-76b1452b3049"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:12.480994 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.480824 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6e5fe5-65ec-431a-9690-ab6b19ec81a5-kube-api-access-p9d4v" pod "aa6e5fe5-65ec-431a-9690-ab6b19ec81a5" (UID: "aa6e5fe5-65ec-431a-9690-ab6b19ec81a5"). InnerVolumeSpecName "kube-api-access-p9d4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:12.489173 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.483845 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68455002-0f90-4682-81ef-5280a8236347-util" pod "68455002-0f90-4682-81ef-5280a8236347" (UID: "68455002-0f90-4682-81ef-5280a8236347"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:12.489173 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.484240 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad8739d-6452-40ba-b7cd-76b1452b3049-kube-api-access-khkx9" pod "5ad8739d-6452-40ba-b7cd-76b1452b3049" (UID: "5ad8739d-6452-40ba-b7cd-76b1452b3049"). InnerVolumeSpecName "kube-api-access-khkx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:12.489173 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.485420 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad8739d-6452-40ba-b7cd-76b1452b3049-util" pod "5ad8739d-6452-40ba-b7cd-76b1452b3049" (UID: "5ad8739d-6452-40ba-b7cd-76b1452b3049"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:12.489173 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.486015 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6e5fe5-65ec-431a-9690-ab6b19ec81a5-util" pod "aa6e5fe5-65ec-431a-9690-ab6b19ec81a5" (UID: "aa6e5fe5-65ec-431a-9690-ab6b19ec81a5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:12.489173 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.487519 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68455002-0f90-4682-81ef-5280a8236347-kube-api-access-lbrj8" pod "68455002-0f90-4682-81ef-5280a8236347" (UID: "68455002-0f90-4682-81ef-5280a8236347"). InnerVolumeSpecName "kube-api-access-lbrj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:12.579594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.579550 4821 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa6e5fe5-65ec-431a-9690-ab6b19ec81a5-util\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:12.579594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.579590 4821 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ad8739d-6452-40ba-b7cd-76b1452b3049-util\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:12.579594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.579601 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9d4v\" (UniqueName: \"kubernetes.io/projected/aa6e5fe5-65ec-431a-9690-ab6b19ec81a5-kube-api-access-p9d4v\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:12.579852 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.579612 4821 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68455002-0f90-4682-81ef-5280a8236347-bundle\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:12.579852 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.579623 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khkx9\" (UniqueName: \"kubernetes.io/projected/5ad8739d-6452-40ba-b7cd-76b1452b3049-kube-api-access-khkx9\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:12.579852 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.579630 4821 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68455002-0f90-4682-81ef-5280a8236347-util\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:12.579852 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.579639 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbrj8\" (UniqueName: \"kubernetes.io/projected/68455002-0f90-4682-81ef-5280a8236347-kube-api-access-lbrj8\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:12.579852 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:12.579650 4821 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ad8739d-6452-40ba-b7cd-76b1452b3049-bundle\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:13.159663 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.159636 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk" Oct 29 22:45:13.159663 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.159646 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72ppwk" event={"ID":"68455002-0f90-4682-81ef-5280a8236347","Type":"ContainerDied","Data":"0a88f197e53a1e1c34d0b8493d76cb40587e9bac429a4a5ab6a204c097abffd7"} Oct 29 22:45:13.160142 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.159702 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a88f197e53a1e1c34d0b8493d76cb40587e9bac429a4a5ab6a204c097abffd7" Oct 29 22:45:13.162386 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.162356 4821 generic.go:356] "Generic (PLEG): container finished" podID="dd7dfc52-f51d-4764-bc34-b7ad01bca72e" containerID="adb55fb573db3b0810f79cfb83079a6949f16118f178cfe3b08f896b36c1c8f2" exitCode=0 Oct 29 22:45:13.162833 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.162775 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd" event={"ID":"dd7dfc52-f51d-4764-bc34-b7ad01bca72e","Type":"ContainerDied","Data":"adb55fb573db3b0810f79cfb83079a6949f16118f178cfe3b08f896b36c1c8f2"} Oct 29 22:45:13.165187 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.165161 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq" event={"ID":"5ad8739d-6452-40ba-b7cd-76b1452b3049","Type":"ContainerDied","Data":"a8a9996d0a3bacae1fba763de3aa6ba6ea66aa0760ff7dd9098982e68019c44c"} Oct 29 22:45:13.165323 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.165217 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kk6bq" Oct 29 22:45:13.165323 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.165222 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8a9996d0a3bacae1fba763de3aa6ba6ea66aa0760ff7dd9098982e68019c44c" Oct 29 22:45:13.167549 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.167501 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg" Oct 29 22:45:13.167549 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.167516 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ddflg" event={"ID":"aa6e5fe5-65ec-431a-9690-ab6b19ec81a5","Type":"ContainerDied","Data":"c881a1e056c556c0167b40e7db3b3ff74a978d4d303b1df3682afbe264b2b09a"} Oct 29 22:45:13.167730 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.167557 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c881a1e056c556c0167b40e7db3b3ff74a978d4d303b1df3682afbe264b2b09a" Oct 29 22:45:13.327319 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.326973 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx" Oct 29 22:45:13.363000 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.362973 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn" Oct 29 22:45:13.379702 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.379464 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s" Oct 29 22:45:13.385235 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.385192 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c85156-c8b2-418f-806b-b9ab7d596348-util\" (UniqueName: \"kubernetes.io/empty-dir/04c85156-c8b2-418f-806b-b9ab7d596348-util\") pod \"04c85156-c8b2-418f-806b-b9ab7d596348\" (UID: \"04c85156-c8b2-418f-806b-b9ab7d596348\") " Oct 29 22:45:13.385353 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.385269 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04c85156-c8b2-418f-806b-b9ab7d596348-bundle\" (UniqueName: \"kubernetes.io/empty-dir/04c85156-c8b2-418f-806b-b9ab7d596348-bundle\") pod \"04c85156-c8b2-418f-806b-b9ab7d596348\" (UID: \"04c85156-c8b2-418f-806b-b9ab7d596348\") " Oct 29 22:45:13.385353 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.385313 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a3e1c21-7250-4d3e-8025-e444bd340e00-kube-api-access-2mc5n\" (UniqueName: \"kubernetes.io/projected/8a3e1c21-7250-4d3e-8025-e444bd340e00-kube-api-access-2mc5n\") pod \"8a3e1c21-7250-4d3e-8025-e444bd340e00\" (UID: \"8a3e1c21-7250-4d3e-8025-e444bd340e00\") " Oct 29 22:45:13.385457 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.385352 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04c85156-c8b2-418f-806b-b9ab7d596348-kube-api-access-q7nwx\" (UniqueName: \"kubernetes.io/projected/04c85156-c8b2-418f-806b-b9ab7d596348-kube-api-access-q7nwx\") pod \"04c85156-c8b2-418f-806b-b9ab7d596348\" (UID: \"04c85156-c8b2-418f-806b-b9ab7d596348\") " Oct 29 22:45:13.385457 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.385395 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51154dd-a714-441a-9f27-254335c1910e-util\" (UniqueName: \"kubernetes.io/empty-dir/b51154dd-a714-441a-9f27-254335c1910e-util\") pod \"b51154dd-a714-441a-9f27-254335c1910e\" (UID: \"b51154dd-a714-441a-9f27-254335c1910e\") " Oct 29 22:45:13.385457 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.385430 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a3e1c21-7250-4d3e-8025-e444bd340e00-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a3e1c21-7250-4d3e-8025-e444bd340e00-bundle\") pod \"8a3e1c21-7250-4d3e-8025-e444bd340e00\" (UID: \"8a3e1c21-7250-4d3e-8025-e444bd340e00\") " Oct 29 22:45:13.385618 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.385463 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b51154dd-a714-441a-9f27-254335c1910e-kube-api-access-lll9r\" (UniqueName: \"kubernetes.io/projected/b51154dd-a714-441a-9f27-254335c1910e-kube-api-access-lll9r\") pod \"b51154dd-a714-441a-9f27-254335c1910e\" (UID: \"b51154dd-a714-441a-9f27-254335c1910e\") " Oct 29 22:45:13.385618 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.385496 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a3e1c21-7250-4d3e-8025-e444bd340e00-util\" (UniqueName: \"kubernetes.io/empty-dir/8a3e1c21-7250-4d3e-8025-e444bd340e00-util\") pod \"8a3e1c21-7250-4d3e-8025-e444bd340e00\" (UID: \"8a3e1c21-7250-4d3e-8025-e444bd340e00\") " Oct 29 22:45:13.385618 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.385556 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51154dd-a714-441a-9f27-254335c1910e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b51154dd-a714-441a-9f27-254335c1910e-bundle\") pod \"b51154dd-a714-441a-9f27-254335c1910e\" (UID: \"b51154dd-a714-441a-9f27-254335c1910e\") " Oct 29 22:45:13.386620 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.386267 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c85156-c8b2-418f-806b-b9ab7d596348-bundle" pod "04c85156-c8b2-418f-806b-b9ab7d596348" (UID: "04c85156-c8b2-418f-806b-b9ab7d596348"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:13.386620 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.386307 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51154dd-a714-441a-9f27-254335c1910e-bundle" pod "b51154dd-a714-441a-9f27-254335c1910e" (UID: "b51154dd-a714-441a-9f27-254335c1910e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:13.386620 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.386387 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3e1c21-7250-4d3e-8025-e444bd340e00-bundle" pod "8a3e1c21-7250-4d3e-8025-e444bd340e00" (UID: "8a3e1c21-7250-4d3e-8025-e444bd340e00"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:13.390212 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.390173 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c85156-c8b2-418f-806b-b9ab7d596348-kube-api-access-q7nwx" pod "04c85156-c8b2-418f-806b-b9ab7d596348" (UID: "04c85156-c8b2-418f-806b-b9ab7d596348"). InnerVolumeSpecName "kube-api-access-q7nwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:13.390409 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.390318 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51154dd-a714-441a-9f27-254335c1910e-kube-api-access-lll9r" pod "b51154dd-a714-441a-9f27-254335c1910e" (UID: "b51154dd-a714-441a-9f27-254335c1910e"). InnerVolumeSpecName "kube-api-access-lll9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:13.391440 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.391410 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c85156-c8b2-418f-806b-b9ab7d596348-util" pod "04c85156-c8b2-418f-806b-b9ab7d596348" (UID: "04c85156-c8b2-418f-806b-b9ab7d596348"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:13.391553 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.391510 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3e1c21-7250-4d3e-8025-e444bd340e00-kube-api-access-2mc5n" pod "8a3e1c21-7250-4d3e-8025-e444bd340e00" (UID: "8a3e1c21-7250-4d3e-8025-e444bd340e00"). InnerVolumeSpecName "kube-api-access-2mc5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:13.392643 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.392621 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3e1c21-7250-4d3e-8025-e444bd340e00-util" pod "8a3e1c21-7250-4d3e-8025-e444bd340e00" (UID: "8a3e1c21-7250-4d3e-8025-e444bd340e00"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:13.393785 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.393762 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51154dd-a714-441a-9f27-254335c1910e-util" pod "b51154dd-a714-441a-9f27-254335c1910e" (UID: "b51154dd-a714-441a-9f27-254335c1910e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:13.486759 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.486688 4821 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04c85156-c8b2-418f-806b-b9ab7d596348-util\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:13.486759 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.486723 4821 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04c85156-c8b2-418f-806b-b9ab7d596348-bundle\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:13.486759 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.486740 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mc5n\" (UniqueName: \"kubernetes.io/projected/8a3e1c21-7250-4d3e-8025-e444bd340e00-kube-api-access-2mc5n\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:13.486759 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.486753 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7nwx\" (UniqueName: \"kubernetes.io/projected/04c85156-c8b2-418f-806b-b9ab7d596348-kube-api-access-q7nwx\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:13.487010 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.486767 4821 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b51154dd-a714-441a-9f27-254335c1910e-util\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:13.487010 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.486780 4821 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a3e1c21-7250-4d3e-8025-e444bd340e00-bundle\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:13.487010 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.486793 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lll9r\" (UniqueName: \"kubernetes.io/projected/b51154dd-a714-441a-9f27-254335c1910e-kube-api-access-lll9r\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:13.487010 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.486805 4821 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a3e1c21-7250-4d3e-8025-e444bd340e00-util\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:13.487010 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:13.486817 4821 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b51154dd-a714-441a-9f27-254335c1910e-bundle\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:14.173659 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:14.173618 4821 generic.go:356] "Generic (PLEG): container finished" podID="49ce4f4e-0072-4a0c-b229-8766cde1bee0" containerID="75d313f5a953e50ddfb8e366af7ede4518478050666331e1cb88125da88b344a" exitCode=0 Oct 29 22:45:14.174021 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:14.173723 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k" event={"ID":"49ce4f4e-0072-4a0c-b229-8766cde1bee0","Type":"ContainerDied","Data":"75d313f5a953e50ddfb8e366af7ede4518478050666331e1cb88125da88b344a"} Oct 29 22:45:14.176266 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:14.176235 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx" event={"ID":"b51154dd-a714-441a-9f27-254335c1910e","Type":"ContainerDied","Data":"68b040955be49ff824a5127666ed54b4e0a25d7c9be922de1a04ba8d6ed1cf3a"} Oct 29 22:45:14.176402 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:14.176288 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68b040955be49ff824a5127666ed54b4e0a25d7c9be922de1a04ba8d6ed1cf3a" Oct 29 22:45:14.176402 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:14.176251 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c84kcx" Oct 29 22:45:14.178242 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:14.178220 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s" Oct 29 22:45:14.178354 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:14.178234 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5pd6s" event={"ID":"04c85156-c8b2-418f-806b-b9ab7d596348","Type":"ContainerDied","Data":"3960ae9622e64fc7ac16c3a129f29eb09a383cc0dcb9c6b8341140aff95bd8f3"} Oct 29 22:45:14.178354 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:14.178280 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3960ae9622e64fc7ac16c3a129f29eb09a383cc0dcb9c6b8341140aff95bd8f3" Oct 29 22:45:14.181113 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:14.181039 4821 generic.go:356] "Generic (PLEG): container finished" podID="379859a0-ae5f-4423-9157-13272247ed50" containerID="5753e0a21662cd56fcdf7491d54741880897f4f94f4b62beb79ed5966aa82dee" exitCode=0 Oct 29 22:45:14.181186 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:14.181131 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6" event={"ID":"379859a0-ae5f-4423-9157-13272247ed50","Type":"ContainerDied","Data":"5753e0a21662cd56fcdf7491d54741880897f4f94f4b62beb79ed5966aa82dee"} Oct 29 22:45:14.183296 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:14.183276 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn" Oct 29 22:45:14.183374 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:14.183290 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjdqhn" event={"ID":"8a3e1c21-7250-4d3e-8025-e444bd340e00","Type":"ContainerDied","Data":"4fb3f72f99f2540fccc1f8099116746afdff1b8e1ba8c6a0fc881825608c33b2"} Oct 29 22:45:14.183374 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:14.183325 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fb3f72f99f2540fccc1f8099116746afdff1b8e1ba8c6a0fc881825608c33b2" Oct 29 22:45:14.301955 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:14.301932 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd" Oct 29 22:45:14.393339 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:14.393306 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd7dfc52-f51d-4764-bc34-b7ad01bca72e-kube-api-access-9qgv5\" (UniqueName: \"kubernetes.io/projected/dd7dfc52-f51d-4764-bc34-b7ad01bca72e-kube-api-access-9qgv5\") pod \"dd7dfc52-f51d-4764-bc34-b7ad01bca72e\" (UID: \"dd7dfc52-f51d-4764-bc34-b7ad01bca72e\") " Oct 29 22:45:14.393509 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:14.393348 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd7dfc52-f51d-4764-bc34-b7ad01bca72e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd7dfc52-f51d-4764-bc34-b7ad01bca72e-bundle\") pod \"dd7dfc52-f51d-4764-bc34-b7ad01bca72e\" (UID: \"dd7dfc52-f51d-4764-bc34-b7ad01bca72e\") " Oct 29 22:45:14.393509 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:14.393373 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd7dfc52-f51d-4764-bc34-b7ad01bca72e-util\" (UniqueName: \"kubernetes.io/empty-dir/dd7dfc52-f51d-4764-bc34-b7ad01bca72e-util\") pod \"dd7dfc52-f51d-4764-bc34-b7ad01bca72e\" (UID: \"dd7dfc52-f51d-4764-bc34-b7ad01bca72e\") " Oct 29 22:45:14.393735 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:14.393713 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7dfc52-f51d-4764-bc34-b7ad01bca72e-bundle" pod "dd7dfc52-f51d-4764-bc34-b7ad01bca72e" (UID: "dd7dfc52-f51d-4764-bc34-b7ad01bca72e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:14.395548 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:14.395517 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7dfc52-f51d-4764-bc34-b7ad01bca72e-kube-api-access-9qgv5" pod "dd7dfc52-f51d-4764-bc34-b7ad01bca72e" (UID: "dd7dfc52-f51d-4764-bc34-b7ad01bca72e"). InnerVolumeSpecName "kube-api-access-9qgv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:14.398918 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:14.398898 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7dfc52-f51d-4764-bc34-b7ad01bca72e-util" pod "dd7dfc52-f51d-4764-bc34-b7ad01bca72e" (UID: "dd7dfc52-f51d-4764-bc34-b7ad01bca72e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:14.494795 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:14.494723 4821 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd7dfc52-f51d-4764-bc34-b7ad01bca72e-util\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:14.494795 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:14.494749 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qgv5\" (UniqueName: \"kubernetes.io/projected/dd7dfc52-f51d-4764-bc34-b7ad01bca72e-kube-api-access-9qgv5\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:14.494795 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:14.494759 4821 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd7dfc52-f51d-4764-bc34-b7ad01bca72e-bundle\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:15.193288 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.193259 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd" Oct 29 22:45:15.193288 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.193269 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449smtd" event={"ID":"dd7dfc52-f51d-4764-bc34-b7ad01bca72e","Type":"ContainerDied","Data":"b46a616740872045ae7a0e25b56cd2f983f80a980928f81d82e2d9fbdd520907"} Oct 29 22:45:15.193728 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.193340 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b46a616740872045ae7a0e25b56cd2f983f80a980928f81d82e2d9fbdd520907" Oct 29 22:45:15.328739 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.328655 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6" Oct 29 22:45:15.337279 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.337259 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k" Oct 29 22:45:15.403952 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.403926 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49ce4f4e-0072-4a0c-b229-8766cde1bee0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/49ce4f4e-0072-4a0c-b229-8766cde1bee0-bundle\") pod \"49ce4f4e-0072-4a0c-b229-8766cde1bee0\" (UID: \"49ce4f4e-0072-4a0c-b229-8766cde1bee0\") " Oct 29 22:45:15.404103 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.403966 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49ce4f4e-0072-4a0c-b229-8766cde1bee0-util\" (UniqueName: \"kubernetes.io/empty-dir/49ce4f4e-0072-4a0c-b229-8766cde1bee0-util\") pod \"49ce4f4e-0072-4a0c-b229-8766cde1bee0\" (UID: \"49ce4f4e-0072-4a0c-b229-8766cde1bee0\") " Oct 29 22:45:15.404103 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.404004 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/379859a0-ae5f-4423-9157-13272247ed50-util\" (UniqueName: \"kubernetes.io/empty-dir/379859a0-ae5f-4423-9157-13272247ed50-util\") pod \"379859a0-ae5f-4423-9157-13272247ed50\" (UID: \"379859a0-ae5f-4423-9157-13272247ed50\") " Oct 29 22:45:15.404103 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.404044 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/379859a0-ae5f-4423-9157-13272247ed50-kube-api-access-jm4jr\" (UniqueName: \"kubernetes.io/projected/379859a0-ae5f-4423-9157-13272247ed50-kube-api-access-jm4jr\") pod \"379859a0-ae5f-4423-9157-13272247ed50\" (UID: \"379859a0-ae5f-4423-9157-13272247ed50\") " Oct 29 22:45:15.404103 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.404072 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ce4f4e-0072-4a0c-b229-8766cde1bee0-kube-api-access-jzfj9\" (UniqueName: \"kubernetes.io/projected/49ce4f4e-0072-4a0c-b229-8766cde1bee0-kube-api-access-jzfj9\") pod \"49ce4f4e-0072-4a0c-b229-8766cde1bee0\" (UID: \"49ce4f4e-0072-4a0c-b229-8766cde1bee0\") " Oct 29 22:45:15.404357 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.404114 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/379859a0-ae5f-4423-9157-13272247ed50-bundle\" (UniqueName: \"kubernetes.io/empty-dir/379859a0-ae5f-4423-9157-13272247ed50-bundle\") pod \"379859a0-ae5f-4423-9157-13272247ed50\" (UID: \"379859a0-ae5f-4423-9157-13272247ed50\") " Oct 29 22:45:15.404730 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.404692 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379859a0-ae5f-4423-9157-13272247ed50-bundle" pod "379859a0-ae5f-4423-9157-13272247ed50" (UID: "379859a0-ae5f-4423-9157-13272247ed50"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:15.404909 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.404877 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ce4f4e-0072-4a0c-b229-8766cde1bee0-bundle" pod "49ce4f4e-0072-4a0c-b229-8766cde1bee0" (UID: "49ce4f4e-0072-4a0c-b229-8766cde1bee0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:15.406886 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.406863 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/379859a0-ae5f-4423-9157-13272247ed50-kube-api-access-jm4jr" pod "379859a0-ae5f-4423-9157-13272247ed50" (UID: "379859a0-ae5f-4423-9157-13272247ed50"). InnerVolumeSpecName "kube-api-access-jm4jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:15.407239 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.407194 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ce4f4e-0072-4a0c-b229-8766cde1bee0-kube-api-access-jzfj9" pod "49ce4f4e-0072-4a0c-b229-8766cde1bee0" (UID: "49ce4f4e-0072-4a0c-b229-8766cde1bee0"). InnerVolumeSpecName "kube-api-access-jzfj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:15.411059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.411038 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ce4f4e-0072-4a0c-b229-8766cde1bee0-util" pod "49ce4f4e-0072-4a0c-b229-8766cde1bee0" (UID: "49ce4f4e-0072-4a0c-b229-8766cde1bee0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:15.411226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.411188 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379859a0-ae5f-4423-9157-13272247ed50-util" pod "379859a0-ae5f-4423-9157-13272247ed50" (UID: "379859a0-ae5f-4423-9157-13272247ed50"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:15.504900 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.504828 4821 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49ce4f4e-0072-4a0c-b229-8766cde1bee0-bundle\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:15.504900 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.504855 4821 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49ce4f4e-0072-4a0c-b229-8766cde1bee0-util\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:15.504900 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.504864 4821 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/379859a0-ae5f-4423-9157-13272247ed50-util\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:15.504900 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.504874 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm4jr\" (UniqueName: \"kubernetes.io/projected/379859a0-ae5f-4423-9157-13272247ed50-kube-api-access-jm4jr\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:15.504900 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.504886 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzfj9\" (UniqueName: \"kubernetes.io/projected/49ce4f4e-0072-4a0c-b229-8766cde1bee0-kube-api-access-jzfj9\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:15.504900 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.504894 4821 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/379859a0-ae5f-4423-9157-13272247ed50-bundle\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:45:15.693213 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693165 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl"] Oct 29 22:45:15.693476 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693459 4821 cpu_manager.go:414] "containerMap: removing container" podUID="5ad8739d-6452-40ba-b7cd-76b1452b3049" containerName="util" Oct 29 22:45:15.693515 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693479 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad8739d-6452-40ba-b7cd-76b1452b3049" containerName="util" Oct 29 22:45:15.693515 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693490 4821 cpu_manager.go:414] "containerMap: removing container" podUID="49ce4f4e-0072-4a0c-b229-8766cde1bee0" containerName="extract" Oct 29 22:45:15.693515 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693495 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ce4f4e-0072-4a0c-b229-8766cde1bee0" containerName="extract" Oct 29 22:45:15.693515 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693504 4821 cpu_manager.go:414] "containerMap: removing container" podUID="dd7dfc52-f51d-4764-bc34-b7ad01bca72e" containerName="util" Oct 29 22:45:15.693515 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693509 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd7dfc52-f51d-4764-bc34-b7ad01bca72e" containerName="util" Oct 29 22:45:15.693515 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693515 4821 cpu_manager.go:414] "containerMap: removing container" podUID="5ad8739d-6452-40ba-b7cd-76b1452b3049" containerName="pull" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693520 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad8739d-6452-40ba-b7cd-76b1452b3049" containerName="pull" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693526 4821 cpu_manager.go:414] "containerMap: removing container" podUID="68455002-0f90-4682-81ef-5280a8236347" containerName="pull" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693530 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68455002-0f90-4682-81ef-5280a8236347" containerName="pull" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693535 4821 cpu_manager.go:414] "containerMap: removing container" podUID="49ce4f4e-0072-4a0c-b229-8766cde1bee0" containerName="pull" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693539 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ce4f4e-0072-4a0c-b229-8766cde1bee0" containerName="pull" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693549 4821 cpu_manager.go:414] "containerMap: removing container" podUID="04c85156-c8b2-418f-806b-b9ab7d596348" containerName="util" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693555 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04c85156-c8b2-418f-806b-b9ab7d596348" containerName="util" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693561 4821 cpu_manager.go:414] "containerMap: removing container" podUID="aa6e5fe5-65ec-431a-9690-ab6b19ec81a5" containerName="pull" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693567 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa6e5fe5-65ec-431a-9690-ab6b19ec81a5" containerName="pull" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693580 4821 cpu_manager.go:414] "containerMap: removing container" podUID="04c85156-c8b2-418f-806b-b9ab7d596348" containerName="extract" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693587 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04c85156-c8b2-418f-806b-b9ab7d596348" containerName="extract" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693595 4821 cpu_manager.go:414] "containerMap: removing container" podUID="8a3e1c21-7250-4d3e-8025-e444bd340e00" containerName="pull" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693599 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a3e1c21-7250-4d3e-8025-e444bd340e00" containerName="pull" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693606 4821 cpu_manager.go:414] "containerMap: removing container" podUID="68455002-0f90-4682-81ef-5280a8236347" containerName="util" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693613 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68455002-0f90-4682-81ef-5280a8236347" containerName="util" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693623 4821 cpu_manager.go:414] "containerMap: removing container" podUID="b51154dd-a714-441a-9f27-254335c1910e" containerName="pull" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693630 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b51154dd-a714-441a-9f27-254335c1910e" containerName="pull" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693642 4821 cpu_manager.go:414] "containerMap: removing container" podUID="04c85156-c8b2-418f-806b-b9ab7d596348" containerName="pull" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693648 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04c85156-c8b2-418f-806b-b9ab7d596348" containerName="pull" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693657 4821 cpu_manager.go:414] "containerMap: removing container" podUID="5ad8739d-6452-40ba-b7cd-76b1452b3049" containerName="extract" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693667 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad8739d-6452-40ba-b7cd-76b1452b3049" containerName="extract" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693679 4821 cpu_manager.go:414] "containerMap: removing container" podUID="49ce4f4e-0072-4a0c-b229-8766cde1bee0" containerName="util" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693685 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ce4f4e-0072-4a0c-b229-8766cde1bee0" containerName="util" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693694 4821 cpu_manager.go:414] "containerMap: removing container" podUID="aa6e5fe5-65ec-431a-9690-ab6b19ec81a5" containerName="util" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693700 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa6e5fe5-65ec-431a-9690-ab6b19ec81a5" containerName="util" Oct 29 22:45:15.693706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693713 4821 cpu_manager.go:414] "containerMap: removing container" podUID="379859a0-ae5f-4423-9157-13272247ed50" containerName="util" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693720 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="379859a0-ae5f-4423-9157-13272247ed50" containerName="util" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693756 4821 cpu_manager.go:414] "containerMap: removing container" podUID="dd7dfc52-f51d-4764-bc34-b7ad01bca72e" containerName="pull" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693763 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd7dfc52-f51d-4764-bc34-b7ad01bca72e" containerName="pull" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693772 4821 cpu_manager.go:414] "containerMap: removing container" podUID="379859a0-ae5f-4423-9157-13272247ed50" containerName="pull" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693778 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="379859a0-ae5f-4423-9157-13272247ed50" containerName="pull" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693788 4821 cpu_manager.go:414] "containerMap: removing container" podUID="68455002-0f90-4682-81ef-5280a8236347" containerName="extract" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693795 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68455002-0f90-4682-81ef-5280a8236347" containerName="extract" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693806 4821 cpu_manager.go:414] "containerMap: removing container" podUID="8a3e1c21-7250-4d3e-8025-e444bd340e00" containerName="util" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693813 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a3e1c21-7250-4d3e-8025-e444bd340e00" containerName="util" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693822 4821 cpu_manager.go:414] "containerMap: removing container" podUID="8a3e1c21-7250-4d3e-8025-e444bd340e00" containerName="extract" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693829 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a3e1c21-7250-4d3e-8025-e444bd340e00" containerName="extract" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693837 4821 cpu_manager.go:414] "containerMap: removing container" podUID="379859a0-ae5f-4423-9157-13272247ed50" containerName="extract" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693844 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="379859a0-ae5f-4423-9157-13272247ed50" containerName="extract" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693856 4821 cpu_manager.go:414] "containerMap: removing container" podUID="b51154dd-a714-441a-9f27-254335c1910e" containerName="extract" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693863 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b51154dd-a714-441a-9f27-254335c1910e" containerName="extract" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693877 4821 cpu_manager.go:414] "containerMap: removing container" podUID="b51154dd-a714-441a-9f27-254335c1910e" containerName="util" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693883 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b51154dd-a714-441a-9f27-254335c1910e" containerName="util" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693892 4821 cpu_manager.go:414] "containerMap: removing container" podUID="aa6e5fe5-65ec-431a-9690-ab6b19ec81a5" containerName="extract" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693900 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa6e5fe5-65ec-431a-9690-ab6b19ec81a5" containerName="extract" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693907 4821 cpu_manager.go:414] "containerMap: removing container" podUID="dd7dfc52-f51d-4764-bc34-b7ad01bca72e" containerName="extract" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693914 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd7dfc52-f51d-4764-bc34-b7ad01bca72e" containerName="extract" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693989 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="04c85156-c8b2-418f-806b-b9ab7d596348" containerName="extract" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.693998 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd7dfc52-f51d-4764-bc34-b7ad01bca72e" containerName="extract" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.694008 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa6e5fe5-65ec-431a-9690-ab6b19ec81a5" containerName="extract" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.694016 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="49ce4f4e-0072-4a0c-b229-8766cde1bee0" containerName="extract" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.694028 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="68455002-0f90-4682-81ef-5280a8236347" containerName="extract" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.694035 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad8739d-6452-40ba-b7cd-76b1452b3049" containerName="extract" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.694044 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="379859a0-ae5f-4423-9157-13272247ed50" containerName="extract" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.694051 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a3e1c21-7250-4d3e-8025-e444bd340e00" containerName="extract" Oct 29 22:45:15.694594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.694060 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="b51154dd-a714-441a-9f27-254335c1910e" containerName="extract" Oct 29 22:45:15.697902 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.697886 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" Oct 29 22:45:15.715846 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.715818 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 29 22:45:15.715942 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.715845 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 29 22:45:15.715942 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.715915 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 29 22:45:15.716258 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.716241 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-6q6mp\"" Oct 29 22:45:15.716335 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.716264 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 29 22:45:15.716335 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.716279 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 29 22:45:15.737819 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.737771 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl"] Oct 29 22:45:15.806118 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.806039 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/01ce70c5-758c-4256-a040-0cdad4ebcef0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-b64xl\" (UID: \"01ce70c5-758c-4256-a040-0cdad4ebcef0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" Oct 29 22:45:15.806118 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.806084 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/01ce70c5-758c-4256-a040-0cdad4ebcef0-metrics-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-b64xl\" (UID: \"01ce70c5-758c-4256-a040-0cdad4ebcef0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" Oct 29 22:45:15.806118 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.806103 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bd4x\" (UniqueName: \"kubernetes.io/projected/01ce70c5-758c-4256-a040-0cdad4ebcef0-kube-api-access-4bd4x\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-b64xl\" (UID: \"01ce70c5-758c-4256-a040-0cdad4ebcef0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" Oct 29 22:45:15.806338 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.806191 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01ce70c5-758c-4256-a040-0cdad4ebcef0-webhook-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-b64xl\" (UID: \"01ce70c5-758c-4256-a040-0cdad4ebcef0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" Oct 29 22:45:15.806338 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.806230 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/01ce70c5-758c-4256-a040-0cdad4ebcef0-manifests\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-b64xl\" (UID: \"01ce70c5-758c-4256-a040-0cdad4ebcef0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" Oct 29 22:45:15.806338 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.806314 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01ce70c5-758c-4256-a040-0cdad4ebcef0-tmp\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-b64xl\" (UID: \"01ce70c5-758c-4256-a040-0cdad4ebcef0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" Oct 29 22:45:15.907417 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.907367 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/01ce70c5-758c-4256-a040-0cdad4ebcef0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-b64xl\" (UID: \"01ce70c5-758c-4256-a040-0cdad4ebcef0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" Oct 29 22:45:15.907533 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.907429 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/01ce70c5-758c-4256-a040-0cdad4ebcef0-metrics-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-b64xl\" (UID: \"01ce70c5-758c-4256-a040-0cdad4ebcef0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" Oct 29 22:45:15.907533 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.907451 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bd4x\" (UniqueName: \"kubernetes.io/projected/01ce70c5-758c-4256-a040-0cdad4ebcef0-kube-api-access-4bd4x\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-b64xl\" (UID: \"01ce70c5-758c-4256-a040-0cdad4ebcef0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" Oct 29 22:45:15.907533 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.907471 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01ce70c5-758c-4256-a040-0cdad4ebcef0-webhook-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-b64xl\" (UID: \"01ce70c5-758c-4256-a040-0cdad4ebcef0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" Oct 29 22:45:15.907533 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.907490 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/01ce70c5-758c-4256-a040-0cdad4ebcef0-manifests\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-b64xl\" (UID: \"01ce70c5-758c-4256-a040-0cdad4ebcef0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" Oct 29 22:45:15.907533 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.907529 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01ce70c5-758c-4256-a040-0cdad4ebcef0-tmp\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-b64xl\" (UID: \"01ce70c5-758c-4256-a040-0cdad4ebcef0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" Oct 29 22:45:15.907954 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.907918 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/01ce70c5-758c-4256-a040-0cdad4ebcef0-manifests\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-b64xl\" (UID: \"01ce70c5-758c-4256-a040-0cdad4ebcef0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" Oct 29 22:45:15.910069 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.910039 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01ce70c5-758c-4256-a040-0cdad4ebcef0-tmp\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-b64xl\" (UID: \"01ce70c5-758c-4256-a040-0cdad4ebcef0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" Oct 29 22:45:15.910172 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.910157 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/01ce70c5-758c-4256-a040-0cdad4ebcef0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-b64xl\" (UID: \"01ce70c5-758c-4256-a040-0cdad4ebcef0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" Oct 29 22:45:15.910253 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.910237 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01ce70c5-758c-4256-a040-0cdad4ebcef0-webhook-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-b64xl\" (UID: \"01ce70c5-758c-4256-a040-0cdad4ebcef0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" Oct 29 22:45:15.910287 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.910256 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/01ce70c5-758c-4256-a040-0cdad4ebcef0-metrics-cert\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-b64xl\" (UID: \"01ce70c5-758c-4256-a040-0cdad4ebcef0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" Oct 29 22:45:15.934286 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:15.934258 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bd4x\" (UniqueName: \"kubernetes.io/projected/01ce70c5-758c-4256-a040-0cdad4ebcef0-kube-api-access-4bd4x\") pod \"opendatahub-operator-controller-manager-855c4cdcb6-b64xl\" (UID: \"01ce70c5-758c-4256-a040-0cdad4ebcef0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" Oct 29 22:45:16.011983 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:16.011949 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" Oct 29 22:45:16.199187 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:16.199158 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k" Oct 29 22:45:16.199546 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:16.199157 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbd89k" event={"ID":"49ce4f4e-0072-4a0c-b229-8766cde1bee0","Type":"ContainerDied","Data":"e77169cdbf254c07c0b6db7c6a8c5ef195165ddb4b793d737fe6d3c8c6b0b491"} Oct 29 22:45:16.199546 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:16.199245 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e77169cdbf254c07c0b6db7c6a8c5ef195165ddb4b793d737fe6d3c8c6b0b491" Oct 29 22:45:16.201358 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:16.201335 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6" event={"ID":"379859a0-ae5f-4423-9157-13272247ed50","Type":"ContainerDied","Data":"17e8c7e8d10a6b959809fcb77ea2127defbdbef5b4a4ceaeacc39e093d7e764e"} Oct 29 22:45:16.201469 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:16.201387 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17e8c7e8d10a6b959809fcb77ea2127defbdbef5b4a4ceaeacc39e093d7e764e" Oct 29 22:45:16.201469 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:16.201398 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644h6zb6" Oct 29 22:45:16.371830 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:45:16.371803 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ce70c5_758c_4256_a040_0cdad4ebcef0.slice/crio-c37c72113fa06f457dd3a51e54a5bdc2268b763935b8bd750b0724f9d9c5d0bb WatchSource:0}: Error finding container c37c72113fa06f457dd3a51e54a5bdc2268b763935b8bd750b0724f9d9c5d0bb: Status 404 returned error can't find the container with id c37c72113fa06f457dd3a51e54a5bdc2268b763935b8bd750b0724f9d9c5d0bb Oct 29 22:45:16.372677 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:16.372626 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl"] Oct 29 22:45:17.210063 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:17.210024 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" event={"ID":"01ce70c5-758c-4256-a040-0cdad4ebcef0","Type":"ContainerStarted","Data":"c37c72113fa06f457dd3a51e54a5bdc2268b763935b8bd750b0724f9d9c5d0bb"} Oct 29 22:45:19.222632 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:19.222596 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" event={"ID":"01ce70c5-758c-4256-a040-0cdad4ebcef0","Type":"ContainerStarted","Data":"e8651dab8a355e23d3e7db0b074b4e73f2f71950e5a8fb03acd45bcc94a6b04c"} Oct 29 22:45:20.227832 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:20.227792 4821 generic.go:356] "Generic (PLEG): container finished" podID="01ce70c5-758c-4256-a040-0cdad4ebcef0" containerID="e8651dab8a355e23d3e7db0b074b4e73f2f71950e5a8fb03acd45bcc94a6b04c" exitCode=0 Oct 29 22:45:20.228256 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:20.227882 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" event={"ID":"01ce70c5-758c-4256-a040-0cdad4ebcef0","Type":"ContainerDied","Data":"e8651dab8a355e23d3e7db0b074b4e73f2f71950e5a8fb03acd45bcc94a6b04c"} Oct 29 22:45:21.109350 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:21.109317 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-b5jnp"] Oct 29 22:45:21.112480 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:21.112465 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b5jnp" Oct 29 22:45:21.139216 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:21.139177 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-b5jnp"] Oct 29 22:45:21.146830 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:21.146811 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0151b544-5067-41f7-b860-42c72bd3fd62-tmp\") pod \"jobset-operator-674cd59dc5-b5jnp\" (UID: \"0151b544-5067-41f7-b860-42c72bd3fd62\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b5jnp" Oct 29 22:45:21.146925 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:21.146842 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hprg5\" (UniqueName: \"kubernetes.io/projected/0151b544-5067-41f7-b860-42c72bd3fd62-kube-api-access-hprg5\") pod \"jobset-operator-674cd59dc5-b5jnp\" (UID: \"0151b544-5067-41f7-b860-42c72bd3fd62\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b5jnp" Oct 29 22:45:21.201492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:21.201468 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-zkbqc\"" Oct 29 22:45:21.202245 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:21.202232 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 29 22:45:21.202298 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:21.202274 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:45:21.233730 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:21.233696 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" event={"ID":"01ce70c5-758c-4256-a040-0cdad4ebcef0","Type":"ContainerStarted","Data":"2daebc9d2eef71bddbbb52328ad758a4e0b151881b1eb5463e92ca9c1a10d8de"} Oct 29 22:45:21.234075 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:21.233827 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" Oct 29 22:45:21.247181 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:21.247162 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hprg5\" (UniqueName: \"kubernetes.io/projected/0151b544-5067-41f7-b860-42c72bd3fd62-kube-api-access-hprg5\") pod \"jobset-operator-674cd59dc5-b5jnp\" (UID: \"0151b544-5067-41f7-b860-42c72bd3fd62\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b5jnp" Oct 29 22:45:21.247265 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:21.247238 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0151b544-5067-41f7-b860-42c72bd3fd62-tmp\") pod \"jobset-operator-674cd59dc5-b5jnp\" (UID: \"0151b544-5067-41f7-b860-42c72bd3fd62\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b5jnp" Oct 29 22:45:21.247561 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:21.247546 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0151b544-5067-41f7-b860-42c72bd3fd62-tmp\") pod \"jobset-operator-674cd59dc5-b5jnp\" (UID: \"0151b544-5067-41f7-b860-42c72bd3fd62\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b5jnp" Oct 29 22:45:21.269806 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:21.269771 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hprg5\" (UniqueName: \"kubernetes.io/projected/0151b544-5067-41f7-b860-42c72bd3fd62-kube-api-access-hprg5\") pod \"jobset-operator-674cd59dc5-b5jnp\" (UID: \"0151b544-5067-41f7-b860-42c72bd3fd62\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b5jnp" Oct 29 22:45:21.321559 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:21.321516 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" podStartSLOduration=2.727288173 podStartE2EDuration="6.32150296s" totalImagesPullingTime="3.13850212s" totalInitContainerRuntime="455.712667ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:21.309868992 +0000 UTC m=+175.250416901" watchObservedRunningTime="2026-10-29 22:45:21.32150296 +0000 UTC m=+175.262050867" Oct 29 22:45:21.436408 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:21.436324 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b5jnp" Oct 29 22:45:21.740907 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:21.740829 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-b5jnp"] Oct 29 22:45:21.744188 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:45:21.744162 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0151b544_5067_41f7_b860_42c72bd3fd62.slice/crio-7886f95ef37e53d3da128f5d0837752a0afe8a320d0601281d03c6f3aa7590cf WatchSource:0}: Error finding container 7886f95ef37e53d3da128f5d0837752a0afe8a320d0601281d03c6f3aa7590cf: Status 404 returned error can't find the container with id 7886f95ef37e53d3da128f5d0837752a0afe8a320d0601281d03c6f3aa7590cf Oct 29 22:45:22.238587 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:22.238553 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b5jnp" event={"ID":"0151b544-5067-41f7-b860-42c72bd3fd62","Type":"ContainerStarted","Data":"7886f95ef37e53d3da128f5d0837752a0afe8a320d0601281d03c6f3aa7590cf"} Oct 29 22:45:23.883259 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:45:23.883231 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:45:24.248427 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:24.248334 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b5jnp" event={"ID":"0151b544-5067-41f7-b860-42c72bd3fd62","Type":"ContainerStarted","Data":"4ef0f91621575260d9222605ae357c60c8ece801bc9873e7e157d6c5a0e0c6f8"} Oct 29 22:45:24.293757 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:24.293700 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-b5jnp" podStartSLOduration=1.159559983 podStartE2EDuration="3.293685412s" totalImagesPullingTime="2.134125429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:24.293348252 +0000 UTC m=+178.233896164" watchObservedRunningTime="2026-10-29 22:45:24.293685412 +0000 UTC m=+178.234233322" Oct 29 22:45:27.026454 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:27.026416 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-t6xnn"] Oct 29 22:45:27.032849 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:27.032829 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-t6xnn" Oct 29 22:45:27.044579 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:27.044558 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-t6xnn"] Oct 29 22:45:27.054365 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:27.054327 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:45:27.054365 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:27.054343 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 29 22:45:27.054494 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:27.054375 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-k7pwt\"" Oct 29 22:45:27.093695 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:27.093668 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/810e7bc4-66c2-4317-9709-094676b2c5b0-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-t6xnn\" (UID: \"810e7bc4-66c2-4317-9709-094676b2c5b0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-t6xnn" Oct 29 22:45:27.093804 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:27.093712 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv9gg\" (UniqueName: \"kubernetes.io/projected/810e7bc4-66c2-4317-9709-094676b2c5b0-kube-api-access-zv9gg\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-t6xnn\" (UID: \"810e7bc4-66c2-4317-9709-094676b2c5b0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-t6xnn" Oct 29 22:45:27.194418 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:27.194385 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/810e7bc4-66c2-4317-9709-094676b2c5b0-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-t6xnn\" (UID: \"810e7bc4-66c2-4317-9709-094676b2c5b0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-t6xnn" Oct 29 22:45:27.194568 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:27.194432 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv9gg\" (UniqueName: \"kubernetes.io/projected/810e7bc4-66c2-4317-9709-094676b2c5b0-kube-api-access-zv9gg\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-t6xnn\" (UID: \"810e7bc4-66c2-4317-9709-094676b2c5b0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-t6xnn" Oct 29 22:45:27.194774 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:27.194752 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/810e7bc4-66c2-4317-9709-094676b2c5b0-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-t6xnn\" (UID: \"810e7bc4-66c2-4317-9709-094676b2c5b0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-t6xnn" Oct 29 22:45:27.217683 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:27.217656 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv9gg\" (UniqueName: \"kubernetes.io/projected/810e7bc4-66c2-4317-9709-094676b2c5b0-kube-api-access-zv9gg\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-t6xnn\" (UID: \"810e7bc4-66c2-4317-9709-094676b2c5b0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-t6xnn" Oct 29 22:45:27.343806 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:27.343718 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-t6xnn" Oct 29 22:45:27.674543 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:27.674509 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-t6xnn"] Oct 29 22:45:27.678302 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:45:27.678270 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810e7bc4_66c2_4317_9709_094676b2c5b0.slice/crio-2e6ee068a96ccc9f722a3de05774476c7697b6a96531f4ebf791b649aa4cfe85 WatchSource:0}: Error finding container 2e6ee068a96ccc9f722a3de05774476c7697b6a96531f4ebf791b649aa4cfe85: Status 404 returned error can't find the container with id 2e6ee068a96ccc9f722a3de05774476c7697b6a96531f4ebf791b649aa4cfe85 Oct 29 22:45:28.266285 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:28.266240 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-t6xnn" event={"ID":"810e7bc4-66c2-4317-9709-094676b2c5b0","Type":"ContainerStarted","Data":"2e6ee068a96ccc9f722a3de05774476c7697b6a96531f4ebf791b649aa4cfe85"} Oct 29 22:45:29.272596 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:29.272560 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-t6xnn" event={"ID":"810e7bc4-66c2-4317-9709-094676b2c5b0","Type":"ContainerStarted","Data":"372140f6010c4eee6f8640b7c33e455cc8c16e4ff3fcb96503802bac8f893e94"} Oct 29 22:45:29.340365 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:29.340270 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-t6xnn" podStartSLOduration=1.9464743960000002 podStartE2EDuration="3.340254189s" totalImagesPullingTime="1.393779793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:29.327327255 +0000 UTC m=+183.267875164" watchObservedRunningTime="2026-10-29 22:45:29.340254189 +0000 UTC m=+183.280802099" Oct 29 22:45:32.241074 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:32.241041 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-855c4cdcb6-b64xl" Oct 29 22:45:32.450590 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:32.450555 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-sn4mm"] Oct 29 22:45:32.453911 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:32.453894 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sn4mm" Oct 29 22:45:32.477866 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:32.477843 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 29 22:45:32.477997 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:32.477893 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 29 22:45:32.477997 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:32.477894 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-2bqnb\"" Oct 29 22:45:32.489485 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:32.489465 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-sn4mm"] Oct 29 22:45:32.538686 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:32.538628 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm4dz\" (UniqueName: \"kubernetes.io/projected/bdeff50a-75a8-4272-9120-47c753cdcb5f-kube-api-access-vm4dz\") pod \"cert-manager-webhook-5dbbbf75cc-sn4mm\" (UID: \"bdeff50a-75a8-4272-9120-47c753cdcb5f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sn4mm" Oct 29 22:45:32.538686 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:32.538672 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bdeff50a-75a8-4272-9120-47c753cdcb5f-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-sn4mm\" (UID: \"bdeff50a-75a8-4272-9120-47c753cdcb5f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sn4mm" Oct 29 22:45:32.639827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:32.639797 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm4dz\" (UniqueName: \"kubernetes.io/projected/bdeff50a-75a8-4272-9120-47c753cdcb5f-kube-api-access-vm4dz\") pod \"cert-manager-webhook-5dbbbf75cc-sn4mm\" (UID: \"bdeff50a-75a8-4272-9120-47c753cdcb5f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sn4mm" Oct 29 22:45:32.639966 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:32.639849 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bdeff50a-75a8-4272-9120-47c753cdcb5f-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-sn4mm\" (UID: \"bdeff50a-75a8-4272-9120-47c753cdcb5f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sn4mm" Oct 29 22:45:32.671835 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:32.671800 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bdeff50a-75a8-4272-9120-47c753cdcb5f-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-sn4mm\" (UID: \"bdeff50a-75a8-4272-9120-47c753cdcb5f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sn4mm" Oct 29 22:45:32.673072 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:32.673051 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm4dz\" (UniqueName: \"kubernetes.io/projected/bdeff50a-75a8-4272-9120-47c753cdcb5f-kube-api-access-vm4dz\") pod \"cert-manager-webhook-5dbbbf75cc-sn4mm\" (UID: \"bdeff50a-75a8-4272-9120-47c753cdcb5f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sn4mm" Oct 29 22:45:32.764126 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:32.764090 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sn4mm" Oct 29 22:45:33.062734 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:33.062602 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-sn4mm"] Oct 29 22:45:33.065130 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:45:33.065104 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdeff50a_75a8_4272_9120_47c753cdcb5f.slice/crio-629622349c4016c146ab1b09bd2ceb67acbf5754ff8c78daf3c00550629d7e10 WatchSource:0}: Error finding container 629622349c4016c146ab1b09bd2ceb67acbf5754ff8c78daf3c00550629d7e10: Status 404 returned error can't find the container with id 629622349c4016c146ab1b09bd2ceb67acbf5754ff8c78daf3c00550629d7e10 Oct 29 22:45:33.292115 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:33.292072 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sn4mm" event={"ID":"bdeff50a-75a8-4272-9120-47c753cdcb5f","Type":"ContainerStarted","Data":"629622349c4016c146ab1b09bd2ceb67acbf5754ff8c78daf3c00550629d7e10"} Oct 29 22:45:36.308015 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:36.307911 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sn4mm" event={"ID":"bdeff50a-75a8-4272-9120-47c753cdcb5f","Type":"ContainerStarted","Data":"40762b7261ed342b2929ec5def7d9377fccb9c28841301bc3fbeda1bf85757a8"} Oct 29 22:45:36.308015 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:36.307983 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sn4mm" Oct 29 22:45:36.349143 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:36.349093 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sn4mm" podStartSLOduration=1.5246801749999999 podStartE2EDuration="4.349077147s" totalImagesPullingTime="2.824396972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:36.347392755 +0000 UTC m=+190.287940665" watchObservedRunningTime="2026-10-29 22:45:36.349077147 +0000 UTC m=+190.289625056" Oct 29 22:45:39.234347 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.234310 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-glsj9"] Oct 29 22:45:39.248525 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.248501 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-glsj9" Oct 29 22:45:39.256623 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.256601 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-glsj9"] Oct 29 22:45:39.266555 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.266536 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-c27vw\"" Oct 29 22:45:39.396387 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.396359 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e06a010-96a0-4f69-81bd-eda46fd7a6cf-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-glsj9\" (UID: \"0e06a010-96a0-4f69-81bd-eda46fd7a6cf\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-glsj9" Oct 29 22:45:39.396387 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.396389 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg7fh\" (UniqueName: \"kubernetes.io/projected/0e06a010-96a0-4f69-81bd-eda46fd7a6cf-kube-api-access-mg7fh\") pod \"cert-manager-cainjector-5c57bbd4fc-glsj9\" (UID: \"0e06a010-96a0-4f69-81bd-eda46fd7a6cf\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-glsj9" Oct 29 22:45:39.497686 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.497602 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e06a010-96a0-4f69-81bd-eda46fd7a6cf-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-glsj9\" (UID: \"0e06a010-96a0-4f69-81bd-eda46fd7a6cf\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-glsj9" Oct 29 22:45:39.497686 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.497640 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg7fh\" (UniqueName: \"kubernetes.io/projected/0e06a010-96a0-4f69-81bd-eda46fd7a6cf-kube-api-access-mg7fh\") pod \"cert-manager-cainjector-5c57bbd4fc-glsj9\" (UID: \"0e06a010-96a0-4f69-81bd-eda46fd7a6cf\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-glsj9" Oct 29 22:45:39.523751 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.523726 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e06a010-96a0-4f69-81bd-eda46fd7a6cf-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-glsj9\" (UID: \"0e06a010-96a0-4f69-81bd-eda46fd7a6cf\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-glsj9" Oct 29 22:45:39.523889 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.523757 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg7fh\" (UniqueName: \"kubernetes.io/projected/0e06a010-96a0-4f69-81bd-eda46fd7a6cf-kube-api-access-mg7fh\") pod \"cert-manager-cainjector-5c57bbd4fc-glsj9\" (UID: \"0e06a010-96a0-4f69-81bd-eda46fd7a6cf\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-glsj9" Oct 29 22:45:39.577713 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.577683 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-glsj9" Oct 29 22:45:39.609406 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.609375 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-qzmkg"] Oct 29 22:45:39.630887 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.630858 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-qzmkg" Oct 29 22:45:39.638992 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.638915 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-qzmkg"] Oct 29 22:45:39.648233 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.647734 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 29 22:45:39.648233 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.647778 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 29 22:45:39.648233 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.647880 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wfqwf\"" Oct 29 22:45:39.699218 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.699165 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgsxk\" (UniqueName: \"kubernetes.io/projected/914a9753-42ce-414a-a2b4-065cfa285cc1-kube-api-access-mgsxk\") pod \"kuadrant-operator-controller-manager-cf8469d7b-qzmkg\" (UID: \"914a9753-42ce-414a-a2b4-065cfa285cc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-qzmkg" Oct 29 22:45:39.699380 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.699234 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/914a9753-42ce-414a-a2b4-065cfa285cc1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-qzmkg\" (UID: \"914a9753-42ce-414a-a2b4-065cfa285cc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-qzmkg" Oct 29 22:45:39.800160 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.800090 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgsxk\" (UniqueName: \"kubernetes.io/projected/914a9753-42ce-414a-a2b4-065cfa285cc1-kube-api-access-mgsxk\") pod \"kuadrant-operator-controller-manager-cf8469d7b-qzmkg\" (UID: \"914a9753-42ce-414a-a2b4-065cfa285cc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-qzmkg" Oct 29 22:45:39.800160 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.800131 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/914a9753-42ce-414a-a2b4-065cfa285cc1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-qzmkg\" (UID: \"914a9753-42ce-414a-a2b4-065cfa285cc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-qzmkg" Oct 29 22:45:39.800565 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.800547 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/914a9753-42ce-414a-a2b4-065cfa285cc1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-qzmkg\" (UID: \"914a9753-42ce-414a-a2b4-065cfa285cc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-qzmkg" Oct 29 22:45:39.830373 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.830343 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgsxk\" (UniqueName: \"kubernetes.io/projected/914a9753-42ce-414a-a2b4-065cfa285cc1-kube-api-access-mgsxk\") pod \"kuadrant-operator-controller-manager-cf8469d7b-qzmkg\" (UID: \"914a9753-42ce-414a-a2b4-065cfa285cc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-qzmkg" Oct 29 22:45:39.921140 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.921116 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-glsj9"] Oct 29 22:45:39.923660 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:45:39.923627 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e06a010_96a0_4f69_81bd_eda46fd7a6cf.slice/crio-08629aeac5ce6b584fa6f114f83575aceb5a9df394619b7d6ed0278481c69da2 WatchSource:0}: Error finding container 08629aeac5ce6b584fa6f114f83575aceb5a9df394619b7d6ed0278481c69da2: Status 404 returned error can't find the container with id 08629aeac5ce6b584fa6f114f83575aceb5a9df394619b7d6ed0278481c69da2 Oct 29 22:45:39.963951 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:39.963929 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-qzmkg" Oct 29 22:45:40.300947 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:40.300914 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-qzmkg"] Oct 29 22:45:40.304738 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:45:40.304705 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914a9753_42ce_414a_a2b4_065cfa285cc1.slice/crio-21aa37e3385a2479e948be24c1c10f13c0377bab7e3322359caf99496522d993 WatchSource:0}: Error finding container 21aa37e3385a2479e948be24c1c10f13c0377bab7e3322359caf99496522d993: Status 404 returned error can't find the container with id 21aa37e3385a2479e948be24c1c10f13c0377bab7e3322359caf99496522d993 Oct 29 22:45:40.326136 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:40.326086 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-glsj9" event={"ID":"0e06a010-96a0-4f69-81bd-eda46fd7a6cf","Type":"ContainerStarted","Data":"4aeec1cc8242ffc4f0f97e0bf7a28462f4cb8e1e931e4249311d813b472b8cad"} Oct 29 22:45:40.326269 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:40.326152 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-glsj9" event={"ID":"0e06a010-96a0-4f69-81bd-eda46fd7a6cf","Type":"ContainerStarted","Data":"08629aeac5ce6b584fa6f114f83575aceb5a9df394619b7d6ed0278481c69da2"} Oct 29 22:45:40.327687 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:40.327651 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-qzmkg" event={"ID":"914a9753-42ce-414a-a2b4-065cfa285cc1","Type":"ContainerStarted","Data":"21aa37e3385a2479e948be24c1c10f13c0377bab7e3322359caf99496522d993"} Oct 29 22:45:40.366523 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:40.366458 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-glsj9" podStartSLOduration=1.366441018 podStartE2EDuration="1.366441018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:40.365381281 +0000 UTC m=+194.305929194" watchObservedRunningTime="2026-10-29 22:45:40.366441018 +0000 UTC m=+194.306988928" Oct 29 22:45:41.782532 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:41.782487 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-b8lbs"] Oct 29 22:45:41.785033 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:41.785004 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-b8lbs" Oct 29 22:45:41.812895 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:41.812719 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-b8lbs"] Oct 29 22:45:41.812895 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:41.812843 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-jxtp7\"" Oct 29 22:45:41.919453 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:41.919408 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e4ac757-56a5-4b36-adce-1ecb1dea6dea-bound-sa-token\") pod \"cert-manager-597b95cb84-b8lbs\" (UID: \"5e4ac757-56a5-4b36-adce-1ecb1dea6dea\") " pod="cert-manager/cert-manager-597b95cb84-b8lbs" Oct 29 22:45:41.919641 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:41.919535 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtsdx\" (UniqueName: \"kubernetes.io/projected/5e4ac757-56a5-4b36-adce-1ecb1dea6dea-kube-api-access-qtsdx\") pod \"cert-manager-597b95cb84-b8lbs\" (UID: \"5e4ac757-56a5-4b36-adce-1ecb1dea6dea\") " pod="cert-manager/cert-manager-597b95cb84-b8lbs" Oct 29 22:45:42.020797 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:42.020755 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtsdx\" (UniqueName: \"kubernetes.io/projected/5e4ac757-56a5-4b36-adce-1ecb1dea6dea-kube-api-access-qtsdx\") pod \"cert-manager-597b95cb84-b8lbs\" (UID: \"5e4ac757-56a5-4b36-adce-1ecb1dea6dea\") " pod="cert-manager/cert-manager-597b95cb84-b8lbs" Oct 29 22:45:42.020976 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:42.020813 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e4ac757-56a5-4b36-adce-1ecb1dea6dea-bound-sa-token\") pod \"cert-manager-597b95cb84-b8lbs\" (UID: \"5e4ac757-56a5-4b36-adce-1ecb1dea6dea\") " pod="cert-manager/cert-manager-597b95cb84-b8lbs" Oct 29 22:45:42.053267 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:42.047101 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e4ac757-56a5-4b36-adce-1ecb1dea6dea-bound-sa-token\") pod \"cert-manager-597b95cb84-b8lbs\" (UID: \"5e4ac757-56a5-4b36-adce-1ecb1dea6dea\") " pod="cert-manager/cert-manager-597b95cb84-b8lbs" Oct 29 22:45:42.053267 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:42.047547 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtsdx\" (UniqueName: \"kubernetes.io/projected/5e4ac757-56a5-4b36-adce-1ecb1dea6dea-kube-api-access-qtsdx\") pod \"cert-manager-597b95cb84-b8lbs\" (UID: \"5e4ac757-56a5-4b36-adce-1ecb1dea6dea\") " pod="cert-manager/cert-manager-597b95cb84-b8lbs" Oct 29 22:45:42.099895 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:42.099859 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-b8lbs" Oct 29 22:45:42.314665 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:42.314580 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sn4mm" Oct 29 22:45:42.412355 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:42.412314 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-b8lbs"] Oct 29 22:45:42.416576 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:45:42.416544 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e4ac757_56a5_4b36_adce_1ecb1dea6dea.slice/crio-0227f1496388cecc4ad8160e8c8138b2bd7832774bcb6778bf0d7de76aaa3434 WatchSource:0}: Error finding container 0227f1496388cecc4ad8160e8c8138b2bd7832774bcb6778bf0d7de76aaa3434: Status 404 returned error can't find the container with id 0227f1496388cecc4ad8160e8c8138b2bd7832774bcb6778bf0d7de76aaa3434 Oct 29 22:45:43.112557 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:43.112520 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-khfmc"] Oct 29 22:45:43.114770 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:43.114747 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-khfmc" Oct 29 22:45:43.149045 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:43.149020 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-xmvgw\"" Oct 29 22:45:43.150704 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:43.150684 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-khfmc"] Oct 29 22:45:43.230534 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:43.230503 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9fnt\" (UniqueName: \"kubernetes.io/projected/28d10fb8-128f-44db-b810-48a734f838c5-kube-api-access-p9fnt\") pod \"limitador-operator-controller-manager-5778bb4449-khfmc\" (UID: \"28d10fb8-128f-44db-b810-48a734f838c5\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-khfmc" Oct 29 22:45:43.331109 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:43.331059 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9fnt\" (UniqueName: \"kubernetes.io/projected/28d10fb8-128f-44db-b810-48a734f838c5-kube-api-access-p9fnt\") pod \"limitador-operator-controller-manager-5778bb4449-khfmc\" (UID: \"28d10fb8-128f-44db-b810-48a734f838c5\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-khfmc" Oct 29 22:45:43.342269 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:43.342237 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-b8lbs" event={"ID":"5e4ac757-56a5-4b36-adce-1ecb1dea6dea","Type":"ContainerStarted","Data":"1b7eba0892c16d4e27c00c8a3a9ce5623636954f21af95d493bb75994c05fc59"} Oct 29 22:45:43.342413 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:43.342278 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-b8lbs" event={"ID":"5e4ac757-56a5-4b36-adce-1ecb1dea6dea","Type":"ContainerStarted","Data":"0227f1496388cecc4ad8160e8c8138b2bd7832774bcb6778bf0d7de76aaa3434"} Oct 29 22:45:43.363330 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:43.363267 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9fnt\" (UniqueName: \"kubernetes.io/projected/28d10fb8-128f-44db-b810-48a734f838c5-kube-api-access-p9fnt\") pod \"limitador-operator-controller-manager-5778bb4449-khfmc\" (UID: \"28d10fb8-128f-44db-b810-48a734f838c5\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-khfmc" Oct 29 22:45:43.392485 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:43.392440 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-b8lbs" podStartSLOduration=2.392425988 podStartE2EDuration="2.392425988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:43.391409726 +0000 UTC m=+197.331957636" watchObservedRunningTime="2026-10-29 22:45:43.392425988 +0000 UTC m=+197.332973898" Oct 29 22:45:43.426436 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:43.426405 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-khfmc" Oct 29 22:45:43.737189 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:43.737153 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-khfmc"] Oct 29 22:45:43.738495 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:45:43.738455 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28d10fb8_128f_44db_b810_48a734f838c5.slice/crio-530b1f2d4f77a52a687dc9a54c82cdc6d28ed303f9ab1d7c08bb2c42dcf1b8a5 WatchSource:0}: Error finding container 530b1f2d4f77a52a687dc9a54c82cdc6d28ed303f9ab1d7c08bb2c42dcf1b8a5: Status 404 returned error can't find the container with id 530b1f2d4f77a52a687dc9a54c82cdc6d28ed303f9ab1d7c08bb2c42dcf1b8a5 Oct 29 22:45:44.349933 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:44.349893 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-khfmc" event={"ID":"28d10fb8-128f-44db-b810-48a734f838c5","Type":"ContainerStarted","Data":"530b1f2d4f77a52a687dc9a54c82cdc6d28ed303f9ab1d7c08bb2c42dcf1b8a5"} Oct 29 22:45:46.129806 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:46.129775 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5fbc7f7f7f-ggn5j"] Oct 29 22:45:46.133015 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:46.132997 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-ggn5j" Oct 29 22:45:46.167613 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:46.167586 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5fbc7f7f7f-ggn5j"] Oct 29 22:45:46.235671 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:46.235627 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 29 22:45:46.235671 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:46.235677 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5rrn8\"" Oct 29 22:45:46.235902 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:46.235797 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 29 22:45:46.257996 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:46.257969 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rmpx\" (UniqueName: \"kubernetes.io/projected/6b28ccf4-c3ec-4048-8e70-8963b25be172-kube-api-access-4rmpx\") pod \"kube-auth-proxy-5fbc7f7f7f-ggn5j\" (UID: \"6b28ccf4-c3ec-4048-8e70-8963b25be172\") " pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-ggn5j" Oct 29 22:45:46.258126 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:46.258015 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b28ccf4-c3ec-4048-8e70-8963b25be172-tls-certs\") pod \"kube-auth-proxy-5fbc7f7f7f-ggn5j\" (UID: \"6b28ccf4-c3ec-4048-8e70-8963b25be172\") " pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-ggn5j" Oct 29 22:45:46.258126 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:46.258049 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b28ccf4-c3ec-4048-8e70-8963b25be172-tmp\") pod \"kube-auth-proxy-5fbc7f7f7f-ggn5j\" (UID: \"6b28ccf4-c3ec-4048-8e70-8963b25be172\") " pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-ggn5j" Oct 29 22:45:46.359436 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:46.359408 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rmpx\" (UniqueName: \"kubernetes.io/projected/6b28ccf4-c3ec-4048-8e70-8963b25be172-kube-api-access-4rmpx\") pod \"kube-auth-proxy-5fbc7f7f7f-ggn5j\" (UID: \"6b28ccf4-c3ec-4048-8e70-8963b25be172\") " pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-ggn5j" Oct 29 22:45:46.359623 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:46.359452 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b28ccf4-c3ec-4048-8e70-8963b25be172-tls-certs\") pod \"kube-auth-proxy-5fbc7f7f7f-ggn5j\" (UID: \"6b28ccf4-c3ec-4048-8e70-8963b25be172\") " pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-ggn5j" Oct 29 22:45:46.359623 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:46.359598 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b28ccf4-c3ec-4048-8e70-8963b25be172-tmp\") pod \"kube-auth-proxy-5fbc7f7f7f-ggn5j\" (UID: \"6b28ccf4-c3ec-4048-8e70-8963b25be172\") " pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-ggn5j" Oct 29 22:45:46.360980 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:46.360949 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-qzmkg" event={"ID":"914a9753-42ce-414a-a2b4-065cfa285cc1","Type":"ContainerStarted","Data":"ff6a32e87907475e63858798385cd1d07a10386d812efd05e2d198678f8327c8"} Oct 29 22:45:46.361146 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:46.361092 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-qzmkg" Oct 29 22:45:46.362369 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:46.362348 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b28ccf4-c3ec-4048-8e70-8963b25be172-tls-certs\") pod \"kube-auth-proxy-5fbc7f7f7f-ggn5j\" (UID: \"6b28ccf4-c3ec-4048-8e70-8963b25be172\") " pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-ggn5j" Oct 29 22:45:46.363481 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:46.363462 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b28ccf4-c3ec-4048-8e70-8963b25be172-tmp\") pod \"kube-auth-proxy-5fbc7f7f7f-ggn5j\" (UID: \"6b28ccf4-c3ec-4048-8e70-8963b25be172\") " pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-ggn5j" Oct 29 22:45:46.386963 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:46.386903 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rmpx\" (UniqueName: \"kubernetes.io/projected/6b28ccf4-c3ec-4048-8e70-8963b25be172-kube-api-access-4rmpx\") pod \"kube-auth-proxy-5fbc7f7f7f-ggn5j\" (UID: \"6b28ccf4-c3ec-4048-8e70-8963b25be172\") " pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-ggn5j" Oct 29 22:45:46.410289 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:46.410244 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-qzmkg" podStartSLOduration=2.004886498 podStartE2EDuration="7.410230661s" totalImagesPullingTime="5.405344163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:46.409474205 +0000 UTC m=+200.350022116" watchObservedRunningTime="2026-10-29 22:45:46.410230661 +0000 UTC m=+200.350778571" Oct 29 22:45:46.446090 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:46.446057 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-ggn5j" Oct 29 22:45:46.752148 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:46.752063 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5fbc7f7f7f-ggn5j"] Oct 29 22:45:46.756431 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:45:46.756383 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b28ccf4_c3ec_4048_8e70_8963b25be172.slice/crio-84acde316fa22fc05db778a081c830f571241722ec641e0916a5640be46ec015 WatchSource:0}: Error finding container 84acde316fa22fc05db778a081c830f571241722ec641e0916a5640be46ec015: Status 404 returned error can't find the container with id 84acde316fa22fc05db778a081c830f571241722ec641e0916a5640be46ec015 Oct 29 22:45:47.370006 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:47.369965 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-ggn5j" event={"ID":"6b28ccf4-c3ec-4048-8e70-8963b25be172","Type":"ContainerStarted","Data":"84acde316fa22fc05db778a081c830f571241722ec641e0916a5640be46ec015"} Oct 29 22:45:49.382405 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:49.382364 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-ggn5j" event={"ID":"6b28ccf4-c3ec-4048-8e70-8963b25be172","Type":"ContainerStarted","Data":"f83ce1bc78d0c2459a83ff4ca903021c0be25dc5e0164a43d3710ed9d633209a"} Oct 29 22:45:49.424342 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:49.424292 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5fbc7f7f7f-ggn5j" podStartSLOduration=1.210642161 podStartE2EDuration="3.424278377s" totalImagesPullingTime="2.213636216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:49.423110294 +0000 UTC m=+203.363658204" watchObservedRunningTime="2026-10-29 22:45:49.424278377 +0000 UTC m=+203.364826284" Oct 29 22:45:50.478762 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:50.478730 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-n8p4p"] Oct 29 22:45:50.482064 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:50.482046 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-n8p4p" Oct 29 22:45:50.496338 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:50.496310 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-hpfpp\"" Oct 29 22:45:50.503322 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:50.503300 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 29 22:45:50.509601 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:50.509582 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-n8p4p"] Oct 29 22:45:50.596651 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:50.596615 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88n6k\" (UniqueName: \"kubernetes.io/projected/47662ab4-2668-46cd-9e6b-bad98897c972-kube-api-access-88n6k\") pod \"dns-operator-controller-manager-8557cbdb84-n8p4p\" (UID: \"47662ab4-2668-46cd-9e6b-bad98897c972\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-n8p4p" Oct 29 22:45:50.697216 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:50.697161 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88n6k\" (UniqueName: \"kubernetes.io/projected/47662ab4-2668-46cd-9e6b-bad98897c972-kube-api-access-88n6k\") pod \"dns-operator-controller-manager-8557cbdb84-n8p4p\" (UID: \"47662ab4-2668-46cd-9e6b-bad98897c972\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-n8p4p" Oct 29 22:45:50.723224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:50.723172 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88n6k\" (UniqueName: \"kubernetes.io/projected/47662ab4-2668-46cd-9e6b-bad98897c972-kube-api-access-88n6k\") pod \"dns-operator-controller-manager-8557cbdb84-n8p4p\" (UID: \"47662ab4-2668-46cd-9e6b-bad98897c972\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-n8p4p" Oct 29 22:45:50.794381 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:50.794310 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-n8p4p" Oct 29 22:45:51.120288 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:51.120264 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-n8p4p"] Oct 29 22:45:51.122848 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:45:51.122819 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47662ab4_2668_46cd_9e6b_bad98897c972.slice/crio-b9eb538dfdc654a7826a6b4dbb8d88703899cb1ea99a857e6c15b5140f590184 WatchSource:0}: Error finding container b9eb538dfdc654a7826a6b4dbb8d88703899cb1ea99a857e6c15b5140f590184: Status 404 returned error can't find the container with id b9eb538dfdc654a7826a6b4dbb8d88703899cb1ea99a857e6c15b5140f590184 Oct 29 22:45:51.391825 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:51.391738 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-n8p4p" event={"ID":"47662ab4-2668-46cd-9e6b-bad98897c972","Type":"ContainerStarted","Data":"b9eb538dfdc654a7826a6b4dbb8d88703899cb1ea99a857e6c15b5140f590184"} Oct 29 22:45:53.402505 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:53.402466 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-n8p4p" event={"ID":"47662ab4-2668-46cd-9e6b-bad98897c972","Type":"ContainerStarted","Data":"a51ada58e152eb285eec12c633259dd7a5127b6e4e77848890c864803e5ebb01"} Oct 29 22:45:53.402891 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:53.402553 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-n8p4p" Oct 29 22:45:53.454974 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:53.454919 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-n8p4p" podStartSLOduration=1.873748563 podStartE2EDuration="3.454900943s" totalImagesPullingTime="1.58115238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:53.453310884 +0000 UTC m=+207.393858795" watchObservedRunningTime="2026-10-29 22:45:53.454900943 +0000 UTC m=+207.395448855" Oct 29 22:45:53.883073 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:45:53.883038 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:45:56.416916 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:56.416876 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-khfmc" event={"ID":"28d10fb8-128f-44db-b810-48a734f838c5","Type":"ContainerStarted","Data":"cc4f2b089a852795f3d67a26d74a89a4fb07b6f4ba23bc75627e74c2caaf3e8b"} Oct 29 22:45:56.417375 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:56.416993 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-khfmc" Oct 29 22:45:56.457592 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:56.457540 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-khfmc" podStartSLOduration=1.773577583 podStartE2EDuration="13.457526991s" totalImagesPullingTime="11.683949408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:56.456052481 +0000 UTC m=+210.396600392" watchObservedRunningTime="2026-10-29 22:45:56.457526991 +0000 UTC m=+210.398074901" Oct 29 22:45:57.372185 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:45:57.372152 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-qzmkg" Oct 29 22:46:04.409218 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:04.409179 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-n8p4p" Oct 29 22:46:07.424006 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:07.423974 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-khfmc" Oct 29 22:46:13.350510 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:13.350475 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-qzmkg"] Oct 29 22:46:13.350948 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:13.350728 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-qzmkg" podUID="914a9753-42ce-414a-a2b4-065cfa285cc1" containerName="manager" containerID="cri-o://ff6a32e87907475e63858798385cd1d07a10386d812efd05e2d198678f8327c8" gracePeriod=10 Oct 29 22:46:13.495319 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:13.495285 4821 generic.go:356] "Generic (PLEG): container finished" podID="914a9753-42ce-414a-a2b4-065cfa285cc1" containerID="ff6a32e87907475e63858798385cd1d07a10386d812efd05e2d198678f8327c8" exitCode=0 Oct 29 22:46:13.495507 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:13.495348 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-qzmkg" event={"ID":"914a9753-42ce-414a-a2b4-065cfa285cc1","Type":"ContainerDied","Data":"ff6a32e87907475e63858798385cd1d07a10386d812efd05e2d198678f8327c8"} Oct 29 22:46:13.593306 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:13.593280 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-qzmkg" Oct 29 22:46:13.679708 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:13.679677 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914a9753-42ce-414a-a2b4-065cfa285cc1-kube-api-access-mgsxk\" (UniqueName: \"kubernetes.io/projected/914a9753-42ce-414a-a2b4-065cfa285cc1-kube-api-access-mgsxk\") pod \"914a9753-42ce-414a-a2b4-065cfa285cc1\" (UID: \"914a9753-42ce-414a-a2b4-065cfa285cc1\") " Oct 29 22:46:13.679873 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:13.679726 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914a9753-42ce-414a-a2b4-065cfa285cc1-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/914a9753-42ce-414a-a2b4-065cfa285cc1-extensions-socket-volume\") pod \"914a9753-42ce-414a-a2b4-065cfa285cc1\" (UID: \"914a9753-42ce-414a-a2b4-065cfa285cc1\") " Oct 29 22:46:13.680225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:13.680181 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914a9753-42ce-414a-a2b4-065cfa285cc1-extensions-socket-volume" pod "914a9753-42ce-414a-a2b4-065cfa285cc1" (UID: "914a9753-42ce-414a-a2b4-065cfa285cc1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:13.681940 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:13.681917 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914a9753-42ce-414a-a2b4-065cfa285cc1-kube-api-access-mgsxk" pod "914a9753-42ce-414a-a2b4-065cfa285cc1" (UID: "914a9753-42ce-414a-a2b4-065cfa285cc1"). InnerVolumeSpecName "kube-api-access-mgsxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:13.781140 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:13.781106 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgsxk\" (UniqueName: \"kubernetes.io/projected/914a9753-42ce-414a-a2b4-065cfa285cc1-kube-api-access-mgsxk\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:46:13.781140 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:13.781135 4821 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/914a9753-42ce-414a-a2b4-065cfa285cc1-extensions-socket-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:46:14.501046 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:14.500959 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-qzmkg" Oct 29 22:46:14.501046 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:14.500991 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-qzmkg" event={"ID":"914a9753-42ce-414a-a2b4-065cfa285cc1","Type":"ContainerDied","Data":"21aa37e3385a2479e948be24c1c10f13c0377bab7e3322359caf99496522d993"} Oct 29 22:46:14.501553 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:14.501057 4821 scope.go:122] "RemoveContainer" containerID="ff6a32e87907475e63858798385cd1d07a10386d812efd05e2d198678f8327c8" Oct 29 22:46:14.567284 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:14.567250 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-qzmkg"] Oct 29 22:46:14.571806 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:14.571785 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-qzmkg"] Oct 29 22:46:16.168980 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:16.168935 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="914a9753-42ce-414a-a2b4-065cfa285cc1" path="/var/lib/kubelet/pods/914a9753-42ce-414a-a2b4-065cfa285cc1/volumes" Oct 29 22:46:23.883098 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:46:23.883062 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:46:33.491324 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.491292 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hg59x"] Oct 29 22:46:33.491779 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.491590 4821 cpu_manager.go:414] "containerMap: removing container" podUID="914a9753-42ce-414a-a2b4-065cfa285cc1" containerName="manager" Oct 29 22:46:33.491779 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.491602 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="914a9753-42ce-414a-a2b4-065cfa285cc1" containerName="manager" Oct 29 22:46:33.491779 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.491664 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="914a9753-42ce-414a-a2b4-065cfa285cc1" containerName="manager" Oct 29 22:46:33.503420 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.503391 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hg59x" Oct 29 22:46:33.515568 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.515541 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hg59x"] Oct 29 22:46:33.525135 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.525108 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nrlgf\"" Oct 29 22:46:33.525305 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.525145 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 29 22:46:33.538806 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.538781 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-rvnlb"] Oct 29 22:46:33.542354 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.542332 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rvnlb" Oct 29 22:46:33.593008 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.592975 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 29 22:46:33.599060 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.599027 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-rvnlb"] Oct 29 22:46:33.620520 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.620497 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 29 22:46:33.643791 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.643761 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hg59x"] Oct 29 22:46:33.646495 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.646470 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d-config-file\") pod \"limitador-limitador-74dfcfb894-hg59x\" (UID: \"3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hg59x" Oct 29 22:46:33.646643 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.646503 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd5nk\" (UniqueName: \"kubernetes.io/projected/4d5895b2-39aa-489e-8c3b-f9bf15b1a58a-kube-api-access-fd5nk\") pod \"kuadrant-console-plugin-66c4fb575d-rvnlb\" (UID: \"4d5895b2-39aa-489e-8c3b-f9bf15b1a58a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rvnlb" Oct 29 22:46:33.646643 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.646529 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d5895b2-39aa-489e-8c3b-f9bf15b1a58a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-rvnlb\" (UID: \"4d5895b2-39aa-489e-8c3b-f9bf15b1a58a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rvnlb" Oct 29 22:46:33.646717 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.646663 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4d5895b2-39aa-489e-8c3b-f9bf15b1a58a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-rvnlb\" (UID: \"4d5895b2-39aa-489e-8c3b-f9bf15b1a58a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rvnlb" Oct 29 22:46:33.646717 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.646699 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwsm7\" (UniqueName: \"kubernetes.io/projected/3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d-kube-api-access-lwsm7\") pod \"limitador-limitador-74dfcfb894-hg59x\" (UID: \"3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hg59x" Oct 29 22:46:33.747630 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.747523 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d-config-file\") pod \"limitador-limitador-74dfcfb894-hg59x\" (UID: \"3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hg59x" Oct 29 22:46:33.747630 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.747572 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd5nk\" (UniqueName: \"kubernetes.io/projected/4d5895b2-39aa-489e-8c3b-f9bf15b1a58a-kube-api-access-fd5nk\") pod \"kuadrant-console-plugin-66c4fb575d-rvnlb\" (UID: \"4d5895b2-39aa-489e-8c3b-f9bf15b1a58a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rvnlb" Oct 29 22:46:33.747630 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.747600 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d5895b2-39aa-489e-8c3b-f9bf15b1a58a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-rvnlb\" (UID: \"4d5895b2-39aa-489e-8c3b-f9bf15b1a58a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rvnlb" Oct 29 22:46:33.748000 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.747648 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4d5895b2-39aa-489e-8c3b-f9bf15b1a58a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-rvnlb\" (UID: \"4d5895b2-39aa-489e-8c3b-f9bf15b1a58a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rvnlb" Oct 29 22:46:33.748000 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.747677 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwsm7\" (UniqueName: \"kubernetes.io/projected/3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d-kube-api-access-lwsm7\") pod \"limitador-limitador-74dfcfb894-hg59x\" (UID: \"3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hg59x" Oct 29 22:46:33.748424 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.748399 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d-config-file\") pod \"limitador-limitador-74dfcfb894-hg59x\" (UID: \"3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hg59x" Oct 29 22:46:33.748565 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.748544 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4d5895b2-39aa-489e-8c3b-f9bf15b1a58a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-rvnlb\" (UID: \"4d5895b2-39aa-489e-8c3b-f9bf15b1a58a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rvnlb" Oct 29 22:46:33.750475 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.750453 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d5895b2-39aa-489e-8c3b-f9bf15b1a58a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-rvnlb\" (UID: \"4d5895b2-39aa-489e-8c3b-f9bf15b1a58a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rvnlb" Oct 29 22:46:33.770468 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.770438 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwsm7\" (UniqueName: \"kubernetes.io/projected/3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d-kube-api-access-lwsm7\") pod \"limitador-limitador-74dfcfb894-hg59x\" (UID: \"3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hg59x" Oct 29 22:46:33.770606 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.770582 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd5nk\" (UniqueName: \"kubernetes.io/projected/4d5895b2-39aa-489e-8c3b-f9bf15b1a58a-kube-api-access-fd5nk\") pod \"kuadrant-console-plugin-66c4fb575d-rvnlb\" (UID: \"4d5895b2-39aa-489e-8c3b-f9bf15b1a58a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rvnlb" Oct 29 22:46:33.818473 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.818448 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hg59x" Oct 29 22:46:33.854846 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:33.854819 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rvnlb" Oct 29 22:46:34.124244 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:34.124218 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hg59x"] Oct 29 22:46:34.128004 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:46:34.127979 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3ce7c8_41fc_4ec4_9425_6feb3a1f713d.slice/crio-5908d7e6dc22b6a052a42772514fb39906caf9e8051e1e1e042d30222d5b62ec WatchSource:0}: Error finding container 5908d7e6dc22b6a052a42772514fb39906caf9e8051e1e1e042d30222d5b62ec: Status 404 returned error can't find the container with id 5908d7e6dc22b6a052a42772514fb39906caf9e8051e1e1e042d30222d5b62ec Oct 29 22:46:34.168061 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:34.168042 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-rvnlb"] Oct 29 22:46:34.169864 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:46:34.169837 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d5895b2_39aa_489e_8c3b_f9bf15b1a58a.slice/crio-0e69d4d19c4d9cc3d77f1983d1415382008dec5782678af496dfed3934d1cd3b WatchSource:0}: Error finding container 0e69d4d19c4d9cc3d77f1983d1415382008dec5782678af496dfed3934d1cd3b: Status 404 returned error can't find the container with id 0e69d4d19c4d9cc3d77f1983d1415382008dec5782678af496dfed3934d1cd3b Oct 29 22:46:34.593295 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:34.593179 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rvnlb" event={"ID":"4d5895b2-39aa-489e-8c3b-f9bf15b1a58a","Type":"ContainerStarted","Data":"0e69d4d19c4d9cc3d77f1983d1415382008dec5782678af496dfed3934d1cd3b"} Oct 29 22:46:34.594487 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:34.594462 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hg59x" event={"ID":"3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d","Type":"ContainerStarted","Data":"5908d7e6dc22b6a052a42772514fb39906caf9e8051e1e1e042d30222d5b62ec"} Oct 29 22:46:34.960954 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:34.960915 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2ghtb"] Oct 29 22:46:34.964576 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:34.964548 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-2ghtb" Oct 29 22:46:34.983443 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:34.983417 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2ghtb"] Oct 29 22:46:34.992465 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:34.992419 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ppj7r\"" Oct 29 22:46:35.060982 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:35.060944 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jbqd\" (UniqueName: \"kubernetes.io/projected/febf5339-42cf-4325-b391-fcbd6b240ed8-kube-api-access-7jbqd\") pod \"authorino-56c6c9ddd5-2ghtb\" (UID: \"febf5339-42cf-4325-b391-fcbd6b240ed8\") " pod="kuadrant-system/authorino-56c6c9ddd5-2ghtb" Oct 29 22:46:35.162263 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:35.162226 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jbqd\" (UniqueName: \"kubernetes.io/projected/febf5339-42cf-4325-b391-fcbd6b240ed8-kube-api-access-7jbqd\") pod \"authorino-56c6c9ddd5-2ghtb\" (UID: \"febf5339-42cf-4325-b391-fcbd6b240ed8\") " pod="kuadrant-system/authorino-56c6c9ddd5-2ghtb" Oct 29 22:46:35.191127 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:35.191092 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jbqd\" (UniqueName: \"kubernetes.io/projected/febf5339-42cf-4325-b391-fcbd6b240ed8-kube-api-access-7jbqd\") pod \"authorino-56c6c9ddd5-2ghtb\" (UID: \"febf5339-42cf-4325-b391-fcbd6b240ed8\") " pod="kuadrant-system/authorino-56c6c9ddd5-2ghtb" Oct 29 22:46:35.279791 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:35.279762 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-2ghtb" Oct 29 22:46:35.600945 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:35.600907 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2ghtb"] Oct 29 22:46:35.604951 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:46:35.604922 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfebf5339_42cf_4325_b391_fcbd6b240ed8.slice/crio-4699a2366413c88410f92ddd8a98852954ca35288b96591dc718f052043c7a5e WatchSource:0}: Error finding container 4699a2366413c88410f92ddd8a98852954ca35288b96591dc718f052043c7a5e: Status 404 returned error can't find the container with id 4699a2366413c88410f92ddd8a98852954ca35288b96591dc718f052043c7a5e Oct 29 22:46:35.607950 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:35.607921 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hg59x" event={"ID":"3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d","Type":"ContainerStarted","Data":"5d5d4f554186620bd0933be8ad1fb27f9a470d0fc1bb96b10faf4855e3878087"} Oct 29 22:46:35.608346 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:35.608323 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-hg59x" Oct 29 22:46:35.648707 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:35.648653 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-hg59x" podStartSLOduration=1.583817298 podStartE2EDuration="2.648640394s" totalImagesPullingTime="1.064823096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:35.647018846 +0000 UTC m=+249.587566757" watchObservedRunningTime="2026-10-29 22:46:35.648640394 +0000 UTC m=+249.589188303" Oct 29 22:46:36.615929 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:36.615881 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-2ghtb" event={"ID":"febf5339-42cf-4325-b391-fcbd6b240ed8","Type":"ContainerStarted","Data":"4699a2366413c88410f92ddd8a98852954ca35288b96591dc718f052043c7a5e"} Oct 29 22:46:37.622971 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:37.622932 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-2ghtb" event={"ID":"febf5339-42cf-4325-b391-fcbd6b240ed8","Type":"ContainerStarted","Data":"b881b8ac0ac83c18c6ca76b6ac053b7c5ab9524c54cd31ac7f8fe84799e7e197"} Oct 29 22:46:37.661108 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:37.661052 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-2ghtb" podStartSLOduration=2.069034759 podStartE2EDuration="3.661036057s" totalImagesPullingTime="1.592001298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:37.659812201 +0000 UTC m=+251.600360116" watchObservedRunningTime="2026-10-29 22:46:37.661036057 +0000 UTC m=+251.601583966" Oct 29 22:46:39.634592 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:39.634554 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rvnlb" event={"ID":"4d5895b2-39aa-489e-8c3b-f9bf15b1a58a","Type":"ContainerStarted","Data":"a76e4493a0b3ee36d8bfe825c7e17ef180e90f6812ee9de6711474bcca563484"} Oct 29 22:46:39.673003 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:39.672950 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rvnlb" podStartSLOduration=1.844989046 podStartE2EDuration="6.672935848s" totalImagesPullingTime="4.827946802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:39.672032799 +0000 UTC m=+253.612580745" watchObservedRunningTime="2026-10-29 22:46:39.672935848 +0000 UTC m=+253.613483758" Oct 29 22:46:46.617353 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:46.617320 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-hg59x" Oct 29 22:46:48.508842 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:48.508806 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hg59x"] Oct 29 22:46:48.509341 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:48.509063 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-hg59x" podUID="3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d" containerName="limitador" containerID="cri-o://5d5d4f554186620bd0933be8ad1fb27f9a470d0fc1bb96b10faf4855e3878087" gracePeriod=30 Oct 29 22:46:49.047133 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:49.047099 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hg59x" Oct 29 22:46:49.186616 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:49.186587 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d-config-file\" (UniqueName: \"kubernetes.io/configmap/3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d-config-file\") pod \"3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d\" (UID: \"3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d\") " Oct 29 22:46:49.186808 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:49.186625 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d-kube-api-access-lwsm7\" (UniqueName: \"kubernetes.io/projected/3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d-kube-api-access-lwsm7\") pod \"3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d\" (UID: \"3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d\") " Oct 29 22:46:49.186930 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:49.186906 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d-config-file" pod "3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d" (UID: "3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:49.188923 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:49.188892 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d-kube-api-access-lwsm7" pod "3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d" (UID: "3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d"). InnerVolumeSpecName "kube-api-access-lwsm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:49.287135 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:49.287102 4821 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d-config-file\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:46:49.287135 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:49.287129 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwsm7\" (UniqueName: \"kubernetes.io/projected/3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d-kube-api-access-lwsm7\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:46:49.682328 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:49.682287 4821 generic.go:356] "Generic (PLEG): container finished" podID="3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d" containerID="5d5d4f554186620bd0933be8ad1fb27f9a470d0fc1bb96b10faf4855e3878087" exitCode=0 Oct 29 22:46:49.682814 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:49.682343 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hg59x" Oct 29 22:46:49.682814 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:49.682367 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hg59x" event={"ID":"3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d","Type":"ContainerDied","Data":"5d5d4f554186620bd0933be8ad1fb27f9a470d0fc1bb96b10faf4855e3878087"} Oct 29 22:46:49.682814 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:49.682415 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hg59x" event={"ID":"3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d","Type":"ContainerDied","Data":"5908d7e6dc22b6a052a42772514fb39906caf9e8051e1e1e042d30222d5b62ec"} Oct 29 22:46:49.682814 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:49.682439 4821 scope.go:122] "RemoveContainer" containerID="5d5d4f554186620bd0933be8ad1fb27f9a470d0fc1bb96b10faf4855e3878087" Oct 29 22:46:49.691034 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:49.691014 4821 scope.go:122] "RemoveContainer" containerID="5d5d4f554186620bd0933be8ad1fb27f9a470d0fc1bb96b10faf4855e3878087" Oct 29 22:46:49.691329 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:46:49.691308 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d5d4f554186620bd0933be8ad1fb27f9a470d0fc1bb96b10faf4855e3878087\": container with ID starting with 5d5d4f554186620bd0933be8ad1fb27f9a470d0fc1bb96b10faf4855e3878087 not found: ID does not exist" containerID="5d5d4f554186620bd0933be8ad1fb27f9a470d0fc1bb96b10faf4855e3878087" Oct 29 22:46:49.691406 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:49.691339 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d5d4f554186620bd0933be8ad1fb27f9a470d0fc1bb96b10faf4855e3878087"} err="failed to get container status \"5d5d4f554186620bd0933be8ad1fb27f9a470d0fc1bb96b10faf4855e3878087\": rpc error: code = NotFound desc = could not find container \"5d5d4f554186620bd0933be8ad1fb27f9a470d0fc1bb96b10faf4855e3878087\": container with ID starting with 5d5d4f554186620bd0933be8ad1fb27f9a470d0fc1bb96b10faf4855e3878087 not found: ID does not exist" Oct 29 22:46:49.737145 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:49.737116 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hg59x"] Oct 29 22:46:49.758066 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:49.758041 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hg59x"] Oct 29 22:46:50.156216 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.156168 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-j26l4"] Oct 29 22:46:50.156480 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.156467 4821 cpu_manager.go:414] "containerMap: removing container" podUID="3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d" containerName="limitador" Oct 29 22:46:50.156527 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.156481 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d" containerName="limitador" Oct 29 22:46:50.156561 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.156534 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d" containerName="limitador" Oct 29 22:46:50.160691 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.160671 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-j26l4" Oct 29 22:46:50.168604 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.168575 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d" path="/var/lib/kubelet/pods/3d3ce7c8-41fc-4ec4-9425-6feb3a1f713d/volumes" Oct 29 22:46:50.182791 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.182760 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 29 22:46:50.182939 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.182908 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 29 22:46:50.183044 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.183007 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 29 22:46:50.183044 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.183030 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-mhcmv\"" Oct 29 22:46:50.183379 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.183364 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-j26l4"] Oct 29 22:46:50.209386 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.209359 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-tlbg9"] Oct 29 22:46:50.212414 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.212398 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tlbg9" Oct 29 22:46:50.235439 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.235414 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-tlbg9"] Oct 29 22:46:50.248974 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.248949 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-2gc8z"] Oct 29 22:46:50.252305 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.252288 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-2gc8z" Oct 29 22:46:50.273996 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.273972 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-2gc8z"] Oct 29 22:46:50.281275 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.281254 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-xf6j5\"" Oct 29 22:46:50.281391 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.281287 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-l4sxb\"" Oct 29 22:46:50.281391 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.281319 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 29 22:46:50.281391 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.281328 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 29 22:46:50.295849 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.295826 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxqpq\" (UniqueName: \"kubernetes.io/projected/8d65af4a-3d99-4d72-94ce-cd7d5aee4182-kube-api-access-xxqpq\") pod \"kserve-controller-manager-776cc66d47-j26l4\" (UID: \"8d65af4a-3d99-4d72-94ce-cd7d5aee4182\") " pod="opendatahub/kserve-controller-manager-776cc66d47-j26l4" Oct 29 22:46:50.295931 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.295874 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d65af4a-3d99-4d72-94ce-cd7d5aee4182-cert\") pod \"kserve-controller-manager-776cc66d47-j26l4\" (UID: \"8d65af4a-3d99-4d72-94ce-cd7d5aee4182\") " pod="opendatahub/kserve-controller-manager-776cc66d47-j26l4" Oct 29 22:46:50.396543 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.396506 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq4kc\" (UniqueName: \"kubernetes.io/projected/b3c0c164-7fb1-4216-8b70-921c970b1c9c-kube-api-access-xq4kc\") pod \"llmisvc-controller-manager-869d5dcc5d-tlbg9\" (UID: \"b3c0c164-7fb1-4216-8b70-921c970b1c9c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tlbg9" Oct 29 22:46:50.396722 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.396563 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3c0c164-7fb1-4216-8b70-921c970b1c9c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-tlbg9\" (UID: \"b3c0c164-7fb1-4216-8b70-921c970b1c9c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tlbg9" Oct 29 22:46:50.396722 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.396641 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxqpq\" (UniqueName: \"kubernetes.io/projected/8d65af4a-3d99-4d72-94ce-cd7d5aee4182-kube-api-access-xxqpq\") pod \"kserve-controller-manager-776cc66d47-j26l4\" (UID: \"8d65af4a-3d99-4d72-94ce-cd7d5aee4182\") " pod="opendatahub/kserve-controller-manager-776cc66d47-j26l4" Oct 29 22:46:50.396722 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.396696 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cacdb56-27c0-4b69-ae05-8a9d2344972d-tls-certs\") pod \"model-serving-api-75bd89fdfc-2gc8z\" (UID: \"9cacdb56-27c0-4b69-ae05-8a9d2344972d\") " pod="opendatahub/model-serving-api-75bd89fdfc-2gc8z" Oct 29 22:46:50.396824 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.396808 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d65af4a-3d99-4d72-94ce-cd7d5aee4182-cert\") pod \"kserve-controller-manager-776cc66d47-j26l4\" (UID: \"8d65af4a-3d99-4d72-94ce-cd7d5aee4182\") " pod="opendatahub/kserve-controller-manager-776cc66d47-j26l4" Oct 29 22:46:50.396862 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.396841 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttjvm\" (UniqueName: \"kubernetes.io/projected/9cacdb56-27c0-4b69-ae05-8a9d2344972d-kube-api-access-ttjvm\") pod \"model-serving-api-75bd89fdfc-2gc8z\" (UID: \"9cacdb56-27c0-4b69-ae05-8a9d2344972d\") " pod="opendatahub/model-serving-api-75bd89fdfc-2gc8z" Oct 29 22:46:50.399375 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.399349 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d65af4a-3d99-4d72-94ce-cd7d5aee4182-cert\") pod \"kserve-controller-manager-776cc66d47-j26l4\" (UID: \"8d65af4a-3d99-4d72-94ce-cd7d5aee4182\") " pod="opendatahub/kserve-controller-manager-776cc66d47-j26l4" Oct 29 22:46:50.424458 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.420368 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxqpq\" (UniqueName: \"kubernetes.io/projected/8d65af4a-3d99-4d72-94ce-cd7d5aee4182-kube-api-access-xxqpq\") pod \"kserve-controller-manager-776cc66d47-j26l4\" (UID: \"8d65af4a-3d99-4d72-94ce-cd7d5aee4182\") " pod="opendatahub/kserve-controller-manager-776cc66d47-j26l4" Oct 29 22:46:50.473668 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.473635 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-j26l4" Oct 29 22:46:50.497932 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.497900 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3c0c164-7fb1-4216-8b70-921c970b1c9c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-tlbg9\" (UID: \"b3c0c164-7fb1-4216-8b70-921c970b1c9c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tlbg9" Oct 29 22:46:50.498065 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.497954 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cacdb56-27c0-4b69-ae05-8a9d2344972d-tls-certs\") pod \"model-serving-api-75bd89fdfc-2gc8z\" (UID: \"9cacdb56-27c0-4b69-ae05-8a9d2344972d\") " pod="opendatahub/model-serving-api-75bd89fdfc-2gc8z" Oct 29 22:46:50.498065 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.497992 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttjvm\" (UniqueName: \"kubernetes.io/projected/9cacdb56-27c0-4b69-ae05-8a9d2344972d-kube-api-access-ttjvm\") pod \"model-serving-api-75bd89fdfc-2gc8z\" (UID: \"9cacdb56-27c0-4b69-ae05-8a9d2344972d\") " pod="opendatahub/model-serving-api-75bd89fdfc-2gc8z" Oct 29 22:46:50.498182 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.498123 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq4kc\" (UniqueName: \"kubernetes.io/projected/b3c0c164-7fb1-4216-8b70-921c970b1c9c-kube-api-access-xq4kc\") pod \"llmisvc-controller-manager-869d5dcc5d-tlbg9\" (UID: \"b3c0c164-7fb1-4216-8b70-921c970b1c9c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tlbg9" Oct 29 22:46:50.500583 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.500555 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3c0c164-7fb1-4216-8b70-921c970b1c9c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-tlbg9\" (UID: \"b3c0c164-7fb1-4216-8b70-921c970b1c9c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tlbg9" Oct 29 22:46:50.500732 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.500705 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cacdb56-27c0-4b69-ae05-8a9d2344972d-tls-certs\") pod \"model-serving-api-75bd89fdfc-2gc8z\" (UID: \"9cacdb56-27c0-4b69-ae05-8a9d2344972d\") " pod="opendatahub/model-serving-api-75bd89fdfc-2gc8z" Oct 29 22:46:50.526623 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.526599 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttjvm\" (UniqueName: \"kubernetes.io/projected/9cacdb56-27c0-4b69-ae05-8a9d2344972d-kube-api-access-ttjvm\") pod \"model-serving-api-75bd89fdfc-2gc8z\" (UID: \"9cacdb56-27c0-4b69-ae05-8a9d2344972d\") " pod="opendatahub/model-serving-api-75bd89fdfc-2gc8z" Oct 29 22:46:50.528702 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.528671 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq4kc\" (UniqueName: \"kubernetes.io/projected/b3c0c164-7fb1-4216-8b70-921c970b1c9c-kube-api-access-xq4kc\") pod \"llmisvc-controller-manager-869d5dcc5d-tlbg9\" (UID: \"b3c0c164-7fb1-4216-8b70-921c970b1c9c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tlbg9" Oct 29 22:46:50.564567 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.564535 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-2gc8z" Oct 29 22:46:50.792452 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.792425 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-j26l4"] Oct 29 22:46:50.793864 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:46:50.793838 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d65af4a_3d99_4d72_94ce_cd7d5aee4182.slice/crio-7728024160b11558a34beab75b03ae56bb382d19b25b879bf3b58b345b138de8 WatchSource:0}: Error finding container 7728024160b11558a34beab75b03ae56bb382d19b25b879bf3b58b345b138de8: Status 404 returned error can't find the container with id 7728024160b11558a34beab75b03ae56bb382d19b25b879bf3b58b345b138de8 Oct 29 22:46:50.823722 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.823696 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tlbg9" Oct 29 22:46:50.868283 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:50.868253 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-2gc8z"] Oct 29 22:46:50.872001 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:46:50.871863 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cacdb56_27c0_4b69_ae05_8a9d2344972d.slice/crio-b1a748ff56b06b6f6ad45d0e726c5995a360e92d0f8cd63801ccee8024ae9caf WatchSource:0}: Error finding container b1a748ff56b06b6f6ad45d0e726c5995a360e92d0f8cd63801ccee8024ae9caf: Status 404 returned error can't find the container with id b1a748ff56b06b6f6ad45d0e726c5995a360e92d0f8cd63801ccee8024ae9caf Oct 29 22:46:51.123879 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:51.123818 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-tlbg9"] Oct 29 22:46:51.125803 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:46:51.125780 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb3c0c164_7fb1_4216_8b70_921c970b1c9c.slice/crio-34092923be3018e3febeb53463831416374c1f4e6804044fc1e7e99d099b1af8 WatchSource:0}: Error finding container 34092923be3018e3febeb53463831416374c1f4e6804044fc1e7e99d099b1af8: Status 404 returned error can't find the container with id 34092923be3018e3febeb53463831416374c1f4e6804044fc1e7e99d099b1af8 Oct 29 22:46:51.699279 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:51.699239 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-2gc8z" event={"ID":"9cacdb56-27c0-4b69-ae05-8a9d2344972d","Type":"ContainerStarted","Data":"b1a748ff56b06b6f6ad45d0e726c5995a360e92d0f8cd63801ccee8024ae9caf"} Oct 29 22:46:51.701001 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:51.700938 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tlbg9" event={"ID":"b3c0c164-7fb1-4216-8b70-921c970b1c9c","Type":"ContainerStarted","Data":"34092923be3018e3febeb53463831416374c1f4e6804044fc1e7e99d099b1af8"} Oct 29 22:46:51.702915 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:51.702885 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-j26l4" event={"ID":"8d65af4a-3d99-4d72-94ce-cd7d5aee4182","Type":"ContainerStarted","Data":"7728024160b11558a34beab75b03ae56bb382d19b25b879bf3b58b345b138de8"} Oct 29 22:46:53.716404 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:53.716369 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-j26l4" event={"ID":"8d65af4a-3d99-4d72-94ce-cd7d5aee4182","Type":"ContainerStarted","Data":"b8c1568b6eff74dea9176916321c0363dfe62d9c9d1946307916924ed41ba0e1"} Oct 29 22:46:53.716934 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:53.716437 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-j26l4" Oct 29 22:46:53.718497 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:53.718470 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-2gc8z" event={"ID":"9cacdb56-27c0-4b69-ae05-8a9d2344972d","Type":"ContainerStarted","Data":"0c6fb885604774f01b5008e1783c3932889e5f6a8259a908d48af4f9780602be"} Oct 29 22:46:53.718634 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:53.718559 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-2gc8z" Oct 29 22:46:53.720239 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:53.720191 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tlbg9" event={"ID":"b3c0c164-7fb1-4216-8b70-921c970b1c9c","Type":"ContainerStarted","Data":"de23d024788b697c8b71bb774d64802d5c6e42a7de37e6446b6f1cbedad3de1d"} Oct 29 22:46:53.720361 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:53.720345 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tlbg9" Oct 29 22:46:53.775426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:53.775240 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-j26l4" podStartSLOduration=1.557585858 podStartE2EDuration="3.775223868s" totalImagesPullingTime="2.21763801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:53.774518848 +0000 UTC m=+267.715066759" watchObservedRunningTime="2026-10-29 22:46:53.775223868 +0000 UTC m=+267.715771770" Oct 29 22:46:53.865865 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:53.865805 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tlbg9" podStartSLOduration=1.800945923 podStartE2EDuration="3.865790418s" totalImagesPullingTime="2.064844495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:53.86394582 +0000 UTC m=+267.804493730" watchObservedRunningTime="2026-10-29 22:46:53.865790418 +0000 UTC m=+267.806338328" Oct 29 22:46:53.883553 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:46:53.883518 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:46:53.908098 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:46:53.908047 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-2gc8z" podStartSLOduration=1.990212388 podStartE2EDuration="3.908031533s" totalImagesPullingTime="1.917819145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:53.906942826 +0000 UTC m=+267.847490726" watchObservedRunningTime="2026-10-29 22:46:53.908031533 +0000 UTC m=+267.848579445" Oct 29 22:47:04.729668 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:47:04.729634 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-2gc8z" Oct 29 22:47:23.883834 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:47:23.883789 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:47:24.727437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:47:24.727405 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tlbg9" Oct 29 22:47:24.730422 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:47:24.730403 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-j26l4" Oct 29 22:47:53.883402 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:47:53.883317 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:48:01.683188 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:01.683155 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lzwrl"] Oct 29 22:48:01.686567 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:01.686549 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lzwrl" Oct 29 22:48:01.705990 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:01.705965 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lzwrl"] Oct 29 22:48:01.706642 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:01.706619 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7htk9\" (UniqueName: \"kubernetes.io/projected/dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b-kube-api-access-7htk9\") pod \"authorino-6f6944bcdd-lzwrl\" (UID: \"dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b\") " pod="kuadrant-system/authorino-6f6944bcdd-lzwrl" Oct 29 22:48:01.807048 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:01.806963 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7htk9\" (UniqueName: \"kubernetes.io/projected/dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b-kube-api-access-7htk9\") pod \"authorino-6f6944bcdd-lzwrl\" (UID: \"dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b\") " pod="kuadrant-system/authorino-6f6944bcdd-lzwrl" Oct 29 22:48:01.829975 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:01.829948 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7htk9\" (UniqueName: \"kubernetes.io/projected/dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b-kube-api-access-7htk9\") pod \"authorino-6f6944bcdd-lzwrl\" (UID: \"dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b\") " pod="kuadrant-system/authorino-6f6944bcdd-lzwrl" Oct 29 22:48:01.890576 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:01.890493 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lzwrl"] Oct 29 22:48:01.890761 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:01.890742 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lzwrl" Oct 29 22:48:02.064384 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:02.064353 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-84768dfb48-bjpl5"] Oct 29 22:48:02.068073 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:02.068049 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84768dfb48-bjpl5" Oct 29 22:48:02.069390 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:02.069369 4821 generic.go:356] "Generic (PLEG): container finished" podID="ee04de80-4a76-4efd-8218-1f613b6130f0" containerID="5b3db4492028207bd923310fa379c656ebdcee2d88250f136d3b8fbfbf90f6eb" exitCode=0 Oct 29 22:48:02.069483 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:02.069450 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" event={"ID":"ee04de80-4a76-4efd-8218-1f613b6130f0","Type":"ContainerDied","Data":"5b3db4492028207bd923310fa379c656ebdcee2d88250f136d3b8fbfbf90f6eb"} Oct 29 22:48:02.069805 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:02.069787 4821 scope.go:122] "RemoveContainer" containerID="5b3db4492028207bd923310fa379c656ebdcee2d88250f136d3b8fbfbf90f6eb" Oct 29 22:48:02.070923 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:02.070898 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:48:02.070995 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:02.070957 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:48:02.070995 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:02.070991 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:48:02.088625 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:02.088604 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84768dfb48-bjpl5"] Oct 29 22:48:02.109911 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:02.109888 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/db3ca2e7-ceb1-4395-8e1f-12899fc748b9-tls-cert\") pod \"authorino-84768dfb48-bjpl5\" (UID: \"db3ca2e7-ceb1-4395-8e1f-12899fc748b9\") " pod="kuadrant-system/authorino-84768dfb48-bjpl5" Oct 29 22:48:02.110017 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:02.109924 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvpkq\" (UniqueName: \"kubernetes.io/projected/db3ca2e7-ceb1-4395-8e1f-12899fc748b9-kube-api-access-zvpkq\") pod \"authorino-84768dfb48-bjpl5\" (UID: \"db3ca2e7-ceb1-4395-8e1f-12899fc748b9\") " pod="kuadrant-system/authorino-84768dfb48-bjpl5" Oct 29 22:48:02.124451 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:02.124430 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 29 22:48:02.199326 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:48:02.199277 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcc7514b_c9c1_4f15_ae7b_3abb24b7f19b.slice/crio-c5bd9b90c7524b71be21aaac16cd9c36468e78d0e652dcb5000b0e1a29927710 WatchSource:0}: Error finding container c5bd9b90c7524b71be21aaac16cd9c36468e78d0e652dcb5000b0e1a29927710: Status 404 returned error can't find the container with id c5bd9b90c7524b71be21aaac16cd9c36468e78d0e652dcb5000b0e1a29927710 Oct 29 22:48:02.199551 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:02.199523 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lzwrl"] Oct 29 22:48:02.211129 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:02.211100 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/db3ca2e7-ceb1-4395-8e1f-12899fc748b9-tls-cert\") pod \"authorino-84768dfb48-bjpl5\" (UID: \"db3ca2e7-ceb1-4395-8e1f-12899fc748b9\") " pod="kuadrant-system/authorino-84768dfb48-bjpl5" Oct 29 22:48:02.211282 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:02.211143 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvpkq\" (UniqueName: \"kubernetes.io/projected/db3ca2e7-ceb1-4395-8e1f-12899fc748b9-kube-api-access-zvpkq\") pod \"authorino-84768dfb48-bjpl5\" (UID: \"db3ca2e7-ceb1-4395-8e1f-12899fc748b9\") " pod="kuadrant-system/authorino-84768dfb48-bjpl5" Oct 29 22:48:02.213885 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:02.213863 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/db3ca2e7-ceb1-4395-8e1f-12899fc748b9-tls-cert\") pod \"authorino-84768dfb48-bjpl5\" (UID: \"db3ca2e7-ceb1-4395-8e1f-12899fc748b9\") " pod="kuadrant-system/authorino-84768dfb48-bjpl5" Oct 29 22:48:02.234994 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:02.234968 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvpkq\" (UniqueName: \"kubernetes.io/projected/db3ca2e7-ceb1-4395-8e1f-12899fc748b9-kube-api-access-zvpkq\") pod \"authorino-84768dfb48-bjpl5\" (UID: \"db3ca2e7-ceb1-4395-8e1f-12899fc748b9\") " pod="kuadrant-system/authorino-84768dfb48-bjpl5" Oct 29 22:48:02.381298 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:02.381264 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84768dfb48-bjpl5" Oct 29 22:48:02.688469 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:02.688446 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84768dfb48-bjpl5"] Oct 29 22:48:02.690796 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:48:02.690770 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb3ca2e7_ceb1_4395_8e1f_12899fc748b9.slice/crio-bdc7522b0e714ca838c4ac03b5892bb4275345ac01398a8b945598a1118a7c9c WatchSource:0}: Error finding container bdc7522b0e714ca838c4ac03b5892bb4275345ac01398a8b945598a1118a7c9c: Status 404 returned error can't find the container with id bdc7522b0e714ca838c4ac03b5892bb4275345ac01398a8b945598a1118a7c9c Oct 29 22:48:03.076556 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:03.076521 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84768dfb48-bjpl5" event={"ID":"db3ca2e7-ceb1-4395-8e1f-12899fc748b9","Type":"ContainerStarted","Data":"bdc7522b0e714ca838c4ac03b5892bb4275345ac01398a8b945598a1118a7c9c"} Oct 29 22:48:03.078556 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:03.078475 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-lzwrl" event={"ID":"dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b","Type":"ContainerStarted","Data":"1656ffe412e2f43b5c8b9db07cd5580c7d0f4fd42d35888a2f8d5f596251fb68"} Oct 29 22:48:03.078556 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:03.078523 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-lzwrl" event={"ID":"dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b","Type":"ContainerStarted","Data":"c5bd9b90c7524b71be21aaac16cd9c36468e78d0e652dcb5000b0e1a29927710"} Oct 29 22:48:03.078556 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:03.078523 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-lzwrl" podUID="dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b" containerName="authorino" containerID="cri-o://1656ffe412e2f43b5c8b9db07cd5580c7d0f4fd42d35888a2f8d5f596251fb68" gracePeriod=30 Oct 29 22:48:03.081181 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:03.081157 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" event={"ID":"ee04de80-4a76-4efd-8218-1f613b6130f0","Type":"ContainerStarted","Data":"793f81a2b9419516eb3438b85da589b1ff45a524145085d189513d912af60bce"} Oct 29 22:48:03.131165 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:03.129045 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-lzwrl" podStartSLOduration=1.475789604 podStartE2EDuration="2.129026198s" totalImagesPullingTime="653.236594ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:03.127741078 +0000 UTC m=+337.068288989" watchObservedRunningTime="2026-10-29 22:48:03.129026198 +0000 UTC m=+337.069574109" Oct 29 22:48:03.311641 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:03.311605 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:48:03.312478 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:03.312450 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lwrzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" start-of-body= Oct 29 22:48:03.312592 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:03.312501 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" podUID="ee04de80-4a76-4efd-8218-1f613b6130f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" Oct 29 22:48:03.354126 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:03.354084 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lzwrl" Oct 29 22:48:03.446747 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:03.446714 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b-kube-api-access-7htk9\" (UniqueName: \"kubernetes.io/projected/dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b-kube-api-access-7htk9\") pod \"dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b\" (UID: \"dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b\") " Oct 29 22:48:03.448917 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:03.448887 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b-kube-api-access-7htk9" pod "dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b" (UID: "dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b"). InnerVolumeSpecName "kube-api-access-7htk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:03.547941 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:03.547886 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7htk9\" (UniqueName: \"kubernetes.io/projected/dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b-kube-api-access-7htk9\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:48:04.087634 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:04.087542 4821 generic.go:356] "Generic (PLEG): container finished" podID="dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b" containerID="1656ffe412e2f43b5c8b9db07cd5580c7d0f4fd42d35888a2f8d5f596251fb68" exitCode=0 Oct 29 22:48:04.087634 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:04.087586 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lzwrl" Oct 29 22:48:04.088151 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:04.087629 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-lzwrl" event={"ID":"dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b","Type":"ContainerDied","Data":"1656ffe412e2f43b5c8b9db07cd5580c7d0f4fd42d35888a2f8d5f596251fb68"} Oct 29 22:48:04.088151 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:04.087672 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-lzwrl" event={"ID":"dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b","Type":"ContainerDied","Data":"c5bd9b90c7524b71be21aaac16cd9c36468e78d0e652dcb5000b0e1a29927710"} Oct 29 22:48:04.088151 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:04.087690 4821 scope.go:122] "RemoveContainer" containerID="1656ffe412e2f43b5c8b9db07cd5580c7d0f4fd42d35888a2f8d5f596251fb68" Oct 29 22:48:04.089847 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:04.089808 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84768dfb48-bjpl5" event={"ID":"db3ca2e7-ceb1-4395-8e1f-12899fc748b9","Type":"ContainerStarted","Data":"71f1270638b35d5e0b74e206cf969873c2f8c5fff26036dc629b00190cf90386"} Oct 29 22:48:04.096649 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:04.096634 4821 scope.go:122] "RemoveContainer" containerID="1656ffe412e2f43b5c8b9db07cd5580c7d0f4fd42d35888a2f8d5f596251fb68" Oct 29 22:48:04.096914 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:48:04.096897 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1656ffe412e2f43b5c8b9db07cd5580c7d0f4fd42d35888a2f8d5f596251fb68\": container with ID starting with 1656ffe412e2f43b5c8b9db07cd5580c7d0f4fd42d35888a2f8d5f596251fb68 not found: ID does not exist" containerID="1656ffe412e2f43b5c8b9db07cd5580c7d0f4fd42d35888a2f8d5f596251fb68" Oct 29 22:48:04.096963 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:04.096923 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1656ffe412e2f43b5c8b9db07cd5580c7d0f4fd42d35888a2f8d5f596251fb68"} err="failed to get container status \"1656ffe412e2f43b5c8b9db07cd5580c7d0f4fd42d35888a2f8d5f596251fb68\": rpc error: code = NotFound desc = could not find container \"1656ffe412e2f43b5c8b9db07cd5580c7d0f4fd42d35888a2f8d5f596251fb68\": container with ID starting with 1656ffe412e2f43b5c8b9db07cd5580c7d0f4fd42d35888a2f8d5f596251fb68 not found: ID does not exist" Oct 29 22:48:04.142335 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:04.142308 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lzwrl"] Oct 29 22:48:04.148572 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:04.148538 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lzwrl"] Oct 29 22:48:04.167891 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:04.167865 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b" path="/var/lib/kubelet/pods/dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b/volumes" Oct 29 22:48:04.185860 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:04.185818 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-84768dfb48-bjpl5" podStartSLOduration=1.8111995379999999 podStartE2EDuration="2.185804604s" totalImagesPullingTime="374.605066ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:04.184298335 +0000 UTC m=+338.124846246" watchObservedRunningTime="2026-10-29 22:48:04.185804604 +0000 UTC m=+338.126352514" Oct 29 22:48:04.212661 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:04.212628 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2ghtb"] Oct 29 22:48:04.212839 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:04.212817 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-2ghtb" podUID="febf5339-42cf-4325-b391-fcbd6b240ed8" containerName="authorino" containerID="cri-o://b881b8ac0ac83c18c6ca76b6ac053b7c5ab9524c54cd31ac7f8fe84799e7e197" gracePeriod=30 Oct 29 22:48:04.311859 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:04.311823 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lwrzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" start-of-body= Oct 29 22:48:04.312030 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:04.311882 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" podUID="ee04de80-4a76-4efd-8218-1f613b6130f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" Oct 29 22:48:04.511875 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:04.511848 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-2ghtb" Oct 29 22:48:04.556891 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:04.556861 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/febf5339-42cf-4325-b391-fcbd6b240ed8-kube-api-access-7jbqd\" (UniqueName: \"kubernetes.io/projected/febf5339-42cf-4325-b391-fcbd6b240ed8-kube-api-access-7jbqd\") pod \"febf5339-42cf-4325-b391-fcbd6b240ed8\" (UID: \"febf5339-42cf-4325-b391-fcbd6b240ed8\") " Oct 29 22:48:04.559112 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:04.559090 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/febf5339-42cf-4325-b391-fcbd6b240ed8-kube-api-access-7jbqd" pod "febf5339-42cf-4325-b391-fcbd6b240ed8" (UID: "febf5339-42cf-4325-b391-fcbd6b240ed8"). InnerVolumeSpecName "kube-api-access-7jbqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:04.657937 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:04.657846 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jbqd\" (UniqueName: \"kubernetes.io/projected/febf5339-42cf-4325-b391-fcbd6b240ed8-kube-api-access-7jbqd\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:48:05.096811 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.096719 4821 generic.go:356] "Generic (PLEG): container finished" podID="febf5339-42cf-4325-b391-fcbd6b240ed8" containerID="b881b8ac0ac83c18c6ca76b6ac053b7c5ab9524c54cd31ac7f8fe84799e7e197" exitCode=0 Oct 29 22:48:05.096811 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.096767 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-2ghtb" Oct 29 22:48:05.097366 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.096803 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-2ghtb" event={"ID":"febf5339-42cf-4325-b391-fcbd6b240ed8","Type":"ContainerDied","Data":"b881b8ac0ac83c18c6ca76b6ac053b7c5ab9524c54cd31ac7f8fe84799e7e197"} Oct 29 22:48:05.097366 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.096850 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-2ghtb" event={"ID":"febf5339-42cf-4325-b391-fcbd6b240ed8","Type":"ContainerDied","Data":"4699a2366413c88410f92ddd8a98852954ca35288b96591dc718f052043c7a5e"} Oct 29 22:48:05.097366 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.096867 4821 scope.go:122] "RemoveContainer" containerID="b881b8ac0ac83c18c6ca76b6ac053b7c5ab9524c54cd31ac7f8fe84799e7e197" Oct 29 22:48:05.103698 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.103676 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rx2tt"] Oct 29 22:48:05.104039 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.104021 4821 cpu_manager.go:414] "containerMap: removing container" podUID="dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b" containerName="authorino" Oct 29 22:48:05.104113 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.104041 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b" containerName="authorino" Oct 29 22:48:05.104113 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.104058 4821 cpu_manager.go:414] "containerMap: removing container" podUID="febf5339-42cf-4325-b391-fcbd6b240ed8" containerName="authorino" Oct 29 22:48:05.104113 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.104063 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="febf5339-42cf-4325-b391-fcbd6b240ed8" containerName="authorino" Oct 29 22:48:05.104253 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.104121 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcc7514b-c9c1-4f15-ae7b-3abb24b7f19b" containerName="authorino" Oct 29 22:48:05.104253 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.104129 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="febf5339-42cf-4325-b391-fcbd6b240ed8" containerName="authorino" Oct 29 22:48:05.106512 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.106492 4821 scope.go:122] "RemoveContainer" containerID="b881b8ac0ac83c18c6ca76b6ac053b7c5ab9524c54cd31ac7f8fe84799e7e197" Oct 29 22:48:05.106791 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:48:05.106772 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b881b8ac0ac83c18c6ca76b6ac053b7c5ab9524c54cd31ac7f8fe84799e7e197\": container with ID starting with b881b8ac0ac83c18c6ca76b6ac053b7c5ab9524c54cd31ac7f8fe84799e7e197 not found: ID does not exist" containerID="b881b8ac0ac83c18c6ca76b6ac053b7c5ab9524c54cd31ac7f8fe84799e7e197" Oct 29 22:48:05.106849 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.106799 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b881b8ac0ac83c18c6ca76b6ac053b7c5ab9524c54cd31ac7f8fe84799e7e197"} err="failed to get container status \"b881b8ac0ac83c18c6ca76b6ac053b7c5ab9524c54cd31ac7f8fe84799e7e197\": rpc error: code = NotFound desc = could not find container \"b881b8ac0ac83c18c6ca76b6ac053b7c5ab9524c54cd31ac7f8fe84799e7e197\": container with ID starting with b881b8ac0ac83c18c6ca76b6ac053b7c5ab9524c54cd31ac7f8fe84799e7e197 not found: ID does not exist" Oct 29 22:48:05.108278 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.108261 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rx2tt" Oct 29 22:48:05.148490 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.148448 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rx2tt"] Oct 29 22:48:05.161494 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.161468 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7ks9\" (UniqueName: \"kubernetes.io/projected/e781f3f8-7c1e-4f60-b8a7-236e97928c29-kube-api-access-v7ks9\") pod \"llmisvc-controller-manager-6466c554d8-rx2tt\" (UID: \"e781f3f8-7c1e-4f60-b8a7-236e97928c29\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rx2tt" Oct 29 22:48:05.161625 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.161499 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e781f3f8-7c1e-4f60-b8a7-236e97928c29-cert\") pod \"llmisvc-controller-manager-6466c554d8-rx2tt\" (UID: \"e781f3f8-7c1e-4f60-b8a7-236e97928c29\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rx2tt" Oct 29 22:48:05.207506 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.207470 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2ghtb"] Oct 29 22:48:05.211561 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.211536 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2ghtb"] Oct 29 22:48:05.262812 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.262777 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7ks9\" (UniqueName: \"kubernetes.io/projected/e781f3f8-7c1e-4f60-b8a7-236e97928c29-kube-api-access-v7ks9\") pod \"llmisvc-controller-manager-6466c554d8-rx2tt\" (UID: \"e781f3f8-7c1e-4f60-b8a7-236e97928c29\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rx2tt" Oct 29 22:48:05.262812 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.262813 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e781f3f8-7c1e-4f60-b8a7-236e97928c29-cert\") pod \"llmisvc-controller-manager-6466c554d8-rx2tt\" (UID: \"e781f3f8-7c1e-4f60-b8a7-236e97928c29\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rx2tt" Oct 29 22:48:05.265454 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.265432 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e781f3f8-7c1e-4f60-b8a7-236e97928c29-cert\") pod \"llmisvc-controller-manager-6466c554d8-rx2tt\" (UID: \"e781f3f8-7c1e-4f60-b8a7-236e97928c29\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rx2tt" Oct 29 22:48:05.286046 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.286021 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7ks9\" (UniqueName: \"kubernetes.io/projected/e781f3f8-7c1e-4f60-b8a7-236e97928c29-kube-api-access-v7ks9\") pod \"llmisvc-controller-manager-6466c554d8-rx2tt\" (UID: \"e781f3f8-7c1e-4f60-b8a7-236e97928c29\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rx2tt" Oct 29 22:48:05.312663 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.312626 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lwrzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" start-of-body= Oct 29 22:48:05.312820 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.312682 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" podUID="ee04de80-4a76-4efd-8218-1f613b6130f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" Oct 29 22:48:05.455999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.455972 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rx2tt" Oct 29 22:48:05.754918 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:05.754896 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rx2tt"] Oct 29 22:48:05.757566 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:48:05.757538 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode781f3f8_7c1e_4f60_b8a7_236e97928c29.slice/crio-3e6cecfd17533bcfbd61b0a680fed40d72eadb978ff9ebd66d864d3d4daec1f8 WatchSource:0}: Error finding container 3e6cecfd17533bcfbd61b0a680fed40d72eadb978ff9ebd66d864d3d4daec1f8: Status 404 returned error can't find the container with id 3e6cecfd17533bcfbd61b0a680fed40d72eadb978ff9ebd66d864d3d4daec1f8 Oct 29 22:48:06.104219 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:06.104171 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rx2tt" event={"ID":"e781f3f8-7c1e-4f60-b8a7-236e97928c29","Type":"ContainerStarted","Data":"3e6cecfd17533bcfbd61b0a680fed40d72eadb978ff9ebd66d864d3d4daec1f8"} Oct 29 22:48:06.169278 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:06.169245 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="febf5339-42cf-4325-b391-fcbd6b240ed8" path="/var/lib/kubelet/pods/febf5339-42cf-4325-b391-fcbd6b240ed8/volumes" Oct 29 22:48:06.312397 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:06.312359 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lwrzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" start-of-body= Oct 29 22:48:06.312552 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:06.312421 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" podUID="ee04de80-4a76-4efd-8218-1f613b6130f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" Oct 29 22:48:07.110174 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:07.110139 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rx2tt" event={"ID":"e781f3f8-7c1e-4f60-b8a7-236e97928c29","Type":"ContainerStarted","Data":"dc40f2fa3c71c14d997c31197820f916c861d1f5ce6f51d03701e3586c28a352"} Oct 29 22:48:07.110598 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:07.110257 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rx2tt" Oct 29 22:48:07.164418 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:07.164355 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rx2tt" podStartSLOduration=1.849362996 podStartE2EDuration="2.164341207s" totalImagesPullingTime="314.978211ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:07.162870097 +0000 UTC m=+341.103418007" watchObservedRunningTime="2026-10-29 22:48:07.164341207 +0000 UTC m=+341.104889117" Oct 29 22:48:07.312145 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:07.312110 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-lwrzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" start-of-body= Oct 29 22:48:07.312321 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:07.312166 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" podUID="ee04de80-4a76-4efd-8218-1f613b6130f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.16:15021/healthz/ready\": dial tcp 10.133.0.16:15021: connect: connection refused" Oct 29 22:48:07.903162 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:07.903110 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:48:08.316005 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:08.315934 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:48:08.317080 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:08.317060 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-lwrzr" Oct 29 22:48:23.883696 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:48:23.883655 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:48:38.117733 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:38.117701 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rx2tt" Oct 29 22:48:38.187438 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:38.187405 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-tlbg9"] Oct 29 22:48:38.187655 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:38.187630 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tlbg9" podUID="b3c0c164-7fb1-4216-8b70-921c970b1c9c" containerName="manager" containerID="cri-o://de23d024788b697c8b71bb774d64802d5c6e42a7de37e6446b6f1cbedad3de1d" gracePeriod=30 Oct 29 22:48:38.425564 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:38.425536 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tlbg9" Oct 29 22:48:38.540871 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:38.540828 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3c0c164-7fb1-4216-8b70-921c970b1c9c-cert\" (UniqueName: \"kubernetes.io/secret/b3c0c164-7fb1-4216-8b70-921c970b1c9c-cert\") pod \"b3c0c164-7fb1-4216-8b70-921c970b1c9c\" (UID: \"b3c0c164-7fb1-4216-8b70-921c970b1c9c\") " Oct 29 22:48:38.541065 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:38.540939 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3c0c164-7fb1-4216-8b70-921c970b1c9c-kube-api-access-xq4kc\" (UniqueName: \"kubernetes.io/projected/b3c0c164-7fb1-4216-8b70-921c970b1c9c-kube-api-access-xq4kc\") pod \"b3c0c164-7fb1-4216-8b70-921c970b1c9c\" (UID: \"b3c0c164-7fb1-4216-8b70-921c970b1c9c\") " Oct 29 22:48:38.543351 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:38.543325 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c0c164-7fb1-4216-8b70-921c970b1c9c-kube-api-access-xq4kc" pod "b3c0c164-7fb1-4216-8b70-921c970b1c9c" (UID: "b3c0c164-7fb1-4216-8b70-921c970b1c9c"). InnerVolumeSpecName "kube-api-access-xq4kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:38.543434 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:38.543375 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3c0c164-7fb1-4216-8b70-921c970b1c9c-cert" pod "b3c0c164-7fb1-4216-8b70-921c970b1c9c" (UID: "b3c0c164-7fb1-4216-8b70-921c970b1c9c"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:38.642180 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:38.642148 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq4kc\" (UniqueName: \"kubernetes.io/projected/b3c0c164-7fb1-4216-8b70-921c970b1c9c-kube-api-access-xq4kc\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:48:38.642180 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:38.642177 4821 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3c0c164-7fb1-4216-8b70-921c970b1c9c-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:48:39.259471 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:39.259440 4821 generic.go:356] "Generic (PLEG): container finished" podID="b3c0c164-7fb1-4216-8b70-921c970b1c9c" containerID="de23d024788b697c8b71bb774d64802d5c6e42a7de37e6446b6f1cbedad3de1d" exitCode=0 Oct 29 22:48:39.259894 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:39.259487 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tlbg9" Oct 29 22:48:39.259894 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:39.259525 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tlbg9" event={"ID":"b3c0c164-7fb1-4216-8b70-921c970b1c9c","Type":"ContainerDied","Data":"de23d024788b697c8b71bb774d64802d5c6e42a7de37e6446b6f1cbedad3de1d"} Oct 29 22:48:39.259894 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:39.259582 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-tlbg9" event={"ID":"b3c0c164-7fb1-4216-8b70-921c970b1c9c","Type":"ContainerDied","Data":"34092923be3018e3febeb53463831416374c1f4e6804044fc1e7e99d099b1af8"} Oct 29 22:48:39.259894 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:39.259607 4821 scope.go:122] "RemoveContainer" containerID="de23d024788b697c8b71bb774d64802d5c6e42a7de37e6446b6f1cbedad3de1d" Oct 29 22:48:39.268505 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:39.268489 4821 scope.go:122] "RemoveContainer" containerID="de23d024788b697c8b71bb774d64802d5c6e42a7de37e6446b6f1cbedad3de1d" Oct 29 22:48:39.268775 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:48:39.268753 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de23d024788b697c8b71bb774d64802d5c6e42a7de37e6446b6f1cbedad3de1d\": container with ID starting with de23d024788b697c8b71bb774d64802d5c6e42a7de37e6446b6f1cbedad3de1d not found: ID does not exist" containerID="de23d024788b697c8b71bb774d64802d5c6e42a7de37e6446b6f1cbedad3de1d" Oct 29 22:48:39.268879 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:39.268784 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de23d024788b697c8b71bb774d64802d5c6e42a7de37e6446b6f1cbedad3de1d"} err="failed to get container status \"de23d024788b697c8b71bb774d64802d5c6e42a7de37e6446b6f1cbedad3de1d\": rpc error: code = NotFound desc = could not find container \"de23d024788b697c8b71bb774d64802d5c6e42a7de37e6446b6f1cbedad3de1d\": container with ID starting with de23d024788b697c8b71bb774d64802d5c6e42a7de37e6446b6f1cbedad3de1d not found: ID does not exist" Oct 29 22:48:39.308732 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:39.308707 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-tlbg9"] Oct 29 22:48:39.320475 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:39.320438 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-tlbg9"] Oct 29 22:48:40.167673 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:48:40.167640 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3c0c164-7fb1-4216-8b70-921c970b1c9c" path="/var/lib/kubelet/pods/b3c0c164-7fb1-4216-8b70-921c970b1c9c/volumes" Oct 29 22:48:53.886066 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:48:53.886012 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:49:23.891339 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:49:23.891302 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:49:50.652700 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:50.652665 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6dd5b9556c-7gmf5"] Oct 29 22:49:50.653158 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:50.652961 4821 cpu_manager.go:414] "containerMap: removing container" podUID="b3c0c164-7fb1-4216-8b70-921c970b1c9c" containerName="manager" Oct 29 22:49:50.653158 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:50.652972 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3c0c164-7fb1-4216-8b70-921c970b1c9c" containerName="manager" Oct 29 22:49:50.653158 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:50.653042 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3c0c164-7fb1-4216-8b70-921c970b1c9c" containerName="manager" Oct 29 22:49:50.655967 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:50.655944 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd5b9556c-7gmf5" Oct 29 22:49:50.674891 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:50.674872 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dd5b9556c-7gmf5"] Oct 29 22:49:50.735459 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:50.735428 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhzxl\" (UniqueName: \"kubernetes.io/projected/37c1f92f-3914-412b-a7aa-0cc023dce2c0-kube-api-access-nhzxl\") pod \"authorino-6dd5b9556c-7gmf5\" (UID: \"37c1f92f-3914-412b-a7aa-0cc023dce2c0\") " pod="kuadrant-system/authorino-6dd5b9556c-7gmf5" Oct 29 22:49:50.735624 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:50.735578 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/37c1f92f-3914-412b-a7aa-0cc023dce2c0-tls-cert\") pod \"authorino-6dd5b9556c-7gmf5\" (UID: \"37c1f92f-3914-412b-a7aa-0cc023dce2c0\") " pod="kuadrant-system/authorino-6dd5b9556c-7gmf5" Oct 29 22:49:50.836901 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:50.836866 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/37c1f92f-3914-412b-a7aa-0cc023dce2c0-tls-cert\") pod \"authorino-6dd5b9556c-7gmf5\" (UID: \"37c1f92f-3914-412b-a7aa-0cc023dce2c0\") " pod="kuadrant-system/authorino-6dd5b9556c-7gmf5" Oct 29 22:49:50.836901 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:50.836906 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhzxl\" (UniqueName: \"kubernetes.io/projected/37c1f92f-3914-412b-a7aa-0cc023dce2c0-kube-api-access-nhzxl\") pod \"authorino-6dd5b9556c-7gmf5\" (UID: \"37c1f92f-3914-412b-a7aa-0cc023dce2c0\") " pod="kuadrant-system/authorino-6dd5b9556c-7gmf5" Oct 29 22:49:50.839465 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:50.839440 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/37c1f92f-3914-412b-a7aa-0cc023dce2c0-tls-cert\") pod \"authorino-6dd5b9556c-7gmf5\" (UID: \"37c1f92f-3914-412b-a7aa-0cc023dce2c0\") " pod="kuadrant-system/authorino-6dd5b9556c-7gmf5" Oct 29 22:49:50.859887 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:50.859864 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhzxl\" (UniqueName: \"kubernetes.io/projected/37c1f92f-3914-412b-a7aa-0cc023dce2c0-kube-api-access-nhzxl\") pod \"authorino-6dd5b9556c-7gmf5\" (UID: \"37c1f92f-3914-412b-a7aa-0cc023dce2c0\") " pod="kuadrant-system/authorino-6dd5b9556c-7gmf5" Oct 29 22:49:50.967192 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:50.967102 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd5b9556c-7gmf5" Oct 29 22:49:51.264971 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:51.264921 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dd5b9556c-7gmf5"] Oct 29 22:49:51.267792 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:49:51.267763 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37c1f92f_3914_412b_a7aa_0cc023dce2c0.slice/crio-6f51448faafd7cc70dc14487d8871345863a95a9b47f16cfd38346f570a9bb04 WatchSource:0}: Error finding container 6f51448faafd7cc70dc14487d8871345863a95a9b47f16cfd38346f570a9bb04: Status 404 returned error can't find the container with id 6f51448faafd7cc70dc14487d8871345863a95a9b47f16cfd38346f570a9bb04 Oct 29 22:49:51.269504 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:51.269486 4821 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:49:51.618011 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:51.617976 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dd5b9556c-7gmf5" event={"ID":"37c1f92f-3914-412b-a7aa-0cc023dce2c0","Type":"ContainerStarted","Data":"6f51448faafd7cc70dc14487d8871345863a95a9b47f16cfd38346f570a9bb04"} Oct 29 22:49:52.626367 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:52.626330 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dd5b9556c-7gmf5" event={"ID":"37c1f92f-3914-412b-a7aa-0cc023dce2c0","Type":"ContainerStarted","Data":"3aa372c99fd328238309f665a052e7c777d83d771f2366e81a7656c90e281f56"} Oct 29 22:49:52.669186 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:52.669108 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6dd5b9556c-7gmf5" podStartSLOduration=2.301181784 podStartE2EDuration="2.669088666s" totalImagesPullingTime="367.906882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:52.66613998 +0000 UTC m=+446.606687891" watchObservedRunningTime="2026-10-29 22:49:52.669088666 +0000 UTC m=+446.609636576" Oct 29 22:49:52.695356 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:52.695320 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-84768dfb48-bjpl5"] Oct 29 22:49:52.695584 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:52.695556 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-84768dfb48-bjpl5" podUID="db3ca2e7-ceb1-4395-8e1f-12899fc748b9" containerName="authorino" containerID="cri-o://71f1270638b35d5e0b74e206cf969873c2f8c5fff26036dc629b00190cf90386" gracePeriod=30 Oct 29 22:49:52.953300 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:52.953273 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84768dfb48-bjpl5" Oct 29 22:49:53.056738 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:53.056706 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db3ca2e7-ceb1-4395-8e1f-12899fc748b9-tls-cert\" (UniqueName: \"kubernetes.io/secret/db3ca2e7-ceb1-4395-8e1f-12899fc748b9-tls-cert\") pod \"db3ca2e7-ceb1-4395-8e1f-12899fc748b9\" (UID: \"db3ca2e7-ceb1-4395-8e1f-12899fc748b9\") " Oct 29 22:49:53.056935 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:53.056771 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db3ca2e7-ceb1-4395-8e1f-12899fc748b9-kube-api-access-zvpkq\" (UniqueName: \"kubernetes.io/projected/db3ca2e7-ceb1-4395-8e1f-12899fc748b9-kube-api-access-zvpkq\") pod \"db3ca2e7-ceb1-4395-8e1f-12899fc748b9\" (UID: \"db3ca2e7-ceb1-4395-8e1f-12899fc748b9\") " Oct 29 22:49:53.059037 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:53.059000 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3ca2e7-ceb1-4395-8e1f-12899fc748b9-kube-api-access-zvpkq" pod "db3ca2e7-ceb1-4395-8e1f-12899fc748b9" (UID: "db3ca2e7-ceb1-4395-8e1f-12899fc748b9"). InnerVolumeSpecName "kube-api-access-zvpkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:49:53.067570 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:53.067533 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3ca2e7-ceb1-4395-8e1f-12899fc748b9-tls-cert" pod "db3ca2e7-ceb1-4395-8e1f-12899fc748b9" (UID: "db3ca2e7-ceb1-4395-8e1f-12899fc748b9"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:49:53.158047 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:53.158009 4821 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/db3ca2e7-ceb1-4395-8e1f-12899fc748b9-tls-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:49:53.158047 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:53.158040 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvpkq\" (UniqueName: \"kubernetes.io/projected/db3ca2e7-ceb1-4395-8e1f-12899fc748b9-kube-api-access-zvpkq\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:49:53.637676 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:53.637639 4821 generic.go:356] "Generic (PLEG): container finished" podID="db3ca2e7-ceb1-4395-8e1f-12899fc748b9" containerID="71f1270638b35d5e0b74e206cf969873c2f8c5fff26036dc629b00190cf90386" exitCode=0 Oct 29 22:49:53.638143 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:53.637704 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84768dfb48-bjpl5" Oct 29 22:49:53.638143 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:53.637718 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84768dfb48-bjpl5" event={"ID":"db3ca2e7-ceb1-4395-8e1f-12899fc748b9","Type":"ContainerDied","Data":"71f1270638b35d5e0b74e206cf969873c2f8c5fff26036dc629b00190cf90386"} Oct 29 22:49:53.638143 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:53.637772 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84768dfb48-bjpl5" event={"ID":"db3ca2e7-ceb1-4395-8e1f-12899fc748b9","Type":"ContainerDied","Data":"bdc7522b0e714ca838c4ac03b5892bb4275345ac01398a8b945598a1118a7c9c"} Oct 29 22:49:53.638143 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:53.637793 4821 scope.go:122] "RemoveContainer" containerID="71f1270638b35d5e0b74e206cf969873c2f8c5fff26036dc629b00190cf90386" Oct 29 22:49:53.648083 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:53.648067 4821 scope.go:122] "RemoveContainer" containerID="71f1270638b35d5e0b74e206cf969873c2f8c5fff26036dc629b00190cf90386" Oct 29 22:49:53.648512 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:49:53.648494 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f1270638b35d5e0b74e206cf969873c2f8c5fff26036dc629b00190cf90386\": container with ID starting with 71f1270638b35d5e0b74e206cf969873c2f8c5fff26036dc629b00190cf90386 not found: ID does not exist" containerID="71f1270638b35d5e0b74e206cf969873c2f8c5fff26036dc629b00190cf90386" Oct 29 22:49:53.648576 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:53.648522 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f1270638b35d5e0b74e206cf969873c2f8c5fff26036dc629b00190cf90386"} err="failed to get container status \"71f1270638b35d5e0b74e206cf969873c2f8c5fff26036dc629b00190cf90386\": rpc error: code = NotFound desc = could not find container \"71f1270638b35d5e0b74e206cf969873c2f8c5fff26036dc629b00190cf90386\": container with ID starting with 71f1270638b35d5e0b74e206cf969873c2f8c5fff26036dc629b00190cf90386 not found: ID does not exist" Oct 29 22:49:53.689784 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:53.689750 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-84768dfb48-bjpl5"] Oct 29 22:49:53.702618 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:53.702588 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-84768dfb48-bjpl5"] Oct 29 22:49:53.887304 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:49:53.887253 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:49:54.169551 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:54.169521 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db3ca2e7-ceb1-4395-8e1f-12899fc748b9" path="/var/lib/kubelet/pods/db3ca2e7-ceb1-4395-8e1f-12899fc748b9/volumes" Oct 29 22:49:55.987543 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:55.987511 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc"] Oct 29 22:49:55.988577 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:55.988549 4821 cpu_manager.go:414] "containerMap: removing container" podUID="db3ca2e7-ceb1-4395-8e1f-12899fc748b9" containerName="authorino" Oct 29 22:49:55.988744 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:55.988730 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db3ca2e7-ceb1-4395-8e1f-12899fc748b9" containerName="authorino" Oct 29 22:49:55.988927 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:55.988916 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="db3ca2e7-ceb1-4395-8e1f-12899fc748b9" containerName="authorino" Oct 29 22:49:55.995737 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:55.995713 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.023177 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.023147 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc"] Oct 29 22:49:56.039554 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.037706 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg"] Oct 29 22:49:56.045447 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.045396 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg"] Oct 29 22:49:56.045447 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.045419 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.082697 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.082663 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.082697 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.082699 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6dcs\" (UniqueName: \"kubernetes.io/projected/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-kube-api-access-d6dcs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.082940 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.082779 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.082940 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.082815 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.082940 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.082846 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.082940 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.082864 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.083110 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.082966 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.083110 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.082995 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.083110 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.083024 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.184352 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.184318 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.184529 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.184387 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.184529 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.184411 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6dcs\" (UniqueName: \"kubernetes.io/projected/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-kube-api-access-d6dcs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.184529 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.184445 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.184529 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.184495 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5tm8\" (UniqueName: \"kubernetes.io/projected/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-kube-api-access-g5tm8\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.184529 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.184521 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.184724 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.184549 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.184759 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.184717 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.184792 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.184737 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.184792 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.184781 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.184899 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.184793 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.184899 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.184818 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.184899 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.184878 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.184999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.184911 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.184999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.184949 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.184999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.184977 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.184999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.184989 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.185185 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.185012 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.185185 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.185044 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.185185 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.185076 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.185185 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.185144 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.185422 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.185260 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.185568 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.185545 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.187267 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.187246 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.187366 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.187269 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.207665 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.207635 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.207777 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.207674 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6dcs\" (UniqueName: \"kubernetes.io/projected/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-kube-api-access-d6dcs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ls7vc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.286482 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.286374 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.286482 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.286425 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.286482 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.286454 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.286772 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.286485 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.286772 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.286589 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.286772 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.286612 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5tm8\" (UniqueName: \"kubernetes.io/projected/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-kube-api-access-g5tm8\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.286772 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.286635 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.286772 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.286658 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.286772 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.286737 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.287067 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.286850 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.287126 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.287098 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.287740 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.287706 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.287884 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.287734 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.287884 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.287866 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.289189 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.289159 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.289349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.289332 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.309412 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.309389 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.309771 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.309746 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5tm8\" (UniqueName: \"kubernetes.io/projected/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-kube-api-access-g5tm8\") pod \"maas-default-gateway-openshift-default-6db586dd6c-k4pqg\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.349746 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.349702 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:56.361597 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.361575 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:56.700414 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.700380 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg"] Oct 29 22:49:56.702779 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.702752 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc"] Oct 29 22:49:56.703945 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:49:56.703919 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5402bc5f_c452_45b0_bfe0_ba0a3fc5d83c.slice/crio-99bd5a8ebf8a02fc78f07b4a310367c2c09427fd4cb9cfe74c160af9527f845a WatchSource:0}: Error finding container 99bd5a8ebf8a02fc78f07b4a310367c2c09427fd4cb9cfe74c160af9527f845a: Status 404 returned error can't find the container with id 99bd5a8ebf8a02fc78f07b4a310367c2c09427fd4cb9cfe74c160af9527f845a Oct 29 22:49:56.705984 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.705951 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:49:56.706084 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.706028 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:49:56.706084 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.706072 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:49:56.716149 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:49:56.716123 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2afb440c_9fff_4ea8_8c9e_f9a3bddc2ffc.slice/crio-71a9d5a4f4554c733fd4f09d222f48a912c86b233ade3e4527160b4776d404e5 WatchSource:0}: Error finding container 71a9d5a4f4554c733fd4f09d222f48a912c86b233ade3e4527160b4776d404e5: Status 404 returned error can't find the container with id 71a9d5a4f4554c733fd4f09d222f48a912c86b233ade3e4527160b4776d404e5 Oct 29 22:49:56.718438 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.718408 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:49:56.718532 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.718483 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:49:56.718532 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:56.718518 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:49:57.667925 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:57.667617 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" event={"ID":"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c","Type":"ContainerStarted","Data":"9cd5b0dd2211876c57e9396d401d7cf2b6664e42a779ae57fb885c397ab046e4"} Oct 29 22:49:57.667925 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:57.667684 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" event={"ID":"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c","Type":"ContainerStarted","Data":"99bd5a8ebf8a02fc78f07b4a310367c2c09427fd4cb9cfe74c160af9527f845a"} Oct 29 22:49:57.669480 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:57.669454 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" event={"ID":"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc","Type":"ContainerStarted","Data":"dd1c23af9e77cd826fcbaecb1bdbd8d9504539921fa31975905e5b086f14f4e8"} Oct 29 22:49:57.669609 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:57.669498 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" event={"ID":"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc","Type":"ContainerStarted","Data":"71a9d5a4f4554c733fd4f09d222f48a912c86b233ade3e4527160b4776d404e5"} Oct 29 22:49:57.792927 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:57.792873 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" podStartSLOduration=2.792859216 podStartE2EDuration="2.792859216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:57.739647681 +0000 UTC m=+451.680195592" watchObservedRunningTime="2026-10-29 22:49:57.792859216 +0000 UTC m=+451.733407126" Oct 29 22:49:57.793282 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:57.793252 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" podStartSLOduration=2.793244578 podStartE2EDuration="2.793244578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:57.791135793 +0000 UTC m=+451.731683704" watchObservedRunningTime="2026-10-29 22:49:57.793244578 +0000 UTC m=+451.733792489" Oct 29 22:49:58.353228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:58.352938 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:49:58.353228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:58.353098 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ls7vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 29 22:49:58.353228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:58.353145 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" podUID="2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 29 22:49:58.365323 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:58.365285 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:49:58.365654 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:58.365634 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k4pqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 29 22:49:58.365793 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:58.365773 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" podUID="5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 29 22:49:59.351751 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:59.351702 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ls7vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 29 22:49:59.352462 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:59.352415 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" podUID="2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 29 22:49:59.362673 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:59.362636 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k4pqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 29 22:49:59.362939 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:49:59.362910 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" podUID="5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 29 22:50:00.352619 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:00.352579 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ls7vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 29 22:50:00.353269 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:00.353237 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" podUID="2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 29 22:50:00.364609 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:00.364578 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k4pqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 29 22:50:00.364872 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:00.364843 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" podUID="5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 29 22:50:01.350673 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:01.350633 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ls7vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 29 22:50:01.350939 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:01.350910 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" podUID="2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 29 22:50:01.364228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:01.363778 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k4pqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 29 22:50:01.364228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:01.363833 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" podUID="5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 29 22:50:02.353686 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:02.353644 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ls7vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 29 22:50:02.354025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:02.353982 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" podUID="2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 29 22:50:02.364730 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:02.364686 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k4pqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 29 22:50:02.365237 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:02.364754 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" podUID="5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 29 22:50:03.350756 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:03.350712 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ls7vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 29 22:50:03.350930 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:03.350786 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" podUID="2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 29 22:50:03.362158 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:03.362126 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k4pqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 29 22:50:03.362254 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:03.362190 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" podUID="5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 29 22:50:04.351102 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:04.351061 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ls7vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 29 22:50:04.351540 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:04.351147 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" podUID="2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 29 22:50:04.362431 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:04.362396 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-k4pqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 29 22:50:04.362513 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:04.362457 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" podUID="5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 29 22:50:05.366531 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:05.366495 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:50:05.367032 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:05.367012 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:50:05.367829 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:05.367809 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:50:05.479698 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:05.479666 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:50:05.480342 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:05.480317 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:50:05.481097 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:05.481058 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 22:50:21.228396 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.228361 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2"] Oct 29 22:50:21.231630 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.231604 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.245846 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.245767 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-639067f9-gw-openshift-default-dockercfg-55574\"" Oct 29 22:50:21.260499 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.260467 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2"] Oct 29 22:50:21.347140 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.347103 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61db0858-830e-4190-8d17-6b0a41497b42-istiod-ca-cert\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.347140 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.347140 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-credential-socket\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.347391 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.347171 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61db0858-830e-4190-8d17-6b0a41497b42-istio-token\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.347391 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.347219 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-istio-data\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.347391 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.347351 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-workload-socket\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.347550 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.347391 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-istio-envoy\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.347550 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.347489 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-workload-certs\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.347628 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.347557 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkw9s\" (UniqueName: \"kubernetes.io/projected/61db0858-830e-4190-8d17-6b0a41497b42-kube-api-access-xkw9s\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.347662 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.347636 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61db0858-830e-4190-8d17-6b0a41497b42-istio-podinfo\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.448637 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.448593 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61db0858-830e-4190-8d17-6b0a41497b42-istiod-ca-cert\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.448926 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.448648 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-credential-socket\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.448926 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.448691 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61db0858-830e-4190-8d17-6b0a41497b42-istio-token\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.448926 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.448731 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-istio-data\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.448926 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.448776 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-workload-socket\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.448926 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.448800 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-istio-envoy\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.448926 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.448841 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-workload-certs\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.448926 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.448869 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkw9s\" (UniqueName: \"kubernetes.io/projected/61db0858-830e-4190-8d17-6b0a41497b42-kube-api-access-xkw9s\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.448926 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.448914 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61db0858-830e-4190-8d17-6b0a41497b42-istio-podinfo\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.449449 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.449113 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-credential-socket\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.449449 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.449149 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-istio-data\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.449449 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.449427 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61db0858-830e-4190-8d17-6b0a41497b42-istiod-ca-cert\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.449558 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.449509 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-workload-socket\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.449718 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.449701 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-workload-certs\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.451948 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.451923 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-istio-envoy\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.452136 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.452116 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61db0858-830e-4190-8d17-6b0a41497b42-istio-podinfo\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.471840 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.471806 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61db0858-830e-4190-8d17-6b0a41497b42-istio-token\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.472622 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.472597 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkw9s\" (UniqueName: \"kubernetes.io/projected/61db0858-830e-4190-8d17-6b0a41497b42-kube-api-access-xkw9s\") pod \"e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.550003 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.549905 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:21.873774 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.873753 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2"] Oct 29 22:50:21.875866 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:50:21.875837 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61db0858_830e_4190_8d17_6b0a41497b42.slice/crio-0e5d932561558f4a232d9dcc9fb1ba0c1bd2876d1be4535b7f0b82f1e2b8b495 WatchSource:0}: Error finding container 0e5d932561558f4a232d9dcc9fb1ba0c1bd2876d1be4535b7f0b82f1e2b8b495: Status 404 returned error can't find the container with id 0e5d932561558f4a232d9dcc9fb1ba0c1bd2876d1be4535b7f0b82f1e2b8b495 Oct 29 22:50:21.877957 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.877923 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:50:21.878049 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.878001 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:50:21.878049 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:21.878042 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:50:22.792276 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:22.792233 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" event={"ID":"61db0858-830e-4190-8d17-6b0a41497b42","Type":"ContainerStarted","Data":"71c6e470859fdc24c85a59997ddb8be30488cc9d3466bc43c00751258366dfd6"} Oct 29 22:50:22.792787 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:22.792293 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" event={"ID":"61db0858-830e-4190-8d17-6b0a41497b42","Type":"ContainerStarted","Data":"0e5d932561558f4a232d9dcc9fb1ba0c1bd2876d1be4535b7f0b82f1e2b8b495"} Oct 29 22:50:22.855373 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:22.855315 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" podStartSLOduration=1.855299913 podStartE2EDuration="1.855299913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:22.852886515 +0000 UTC m=+476.793434450" watchObservedRunningTime="2026-10-29 22:50:22.855299913 +0000 UTC m=+476.795847820" Oct 29 22:50:23.550528 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:23.550490 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:23.555646 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:23.555621 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:23.797905 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:23.797877 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:23.798756 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:23.798738 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:23.884950 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:50:23.884909 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:50:29.952465 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:29.952426 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg"] Oct 29 22:50:29.958464 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:29.958425 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg" Oct 29 22:50:29.970991 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:29.970950 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg"] Oct 29 22:50:29.978246 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:29.978222 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1db2136c-dockercfg-xpgts\"" Oct 29 22:50:29.978428 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:29.978231 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1db2136c\"" Oct 29 22:50:30.055415 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:30.055380 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b81dd7c-d225-4645-a679-53f0c1a4d6c3-plugins-config-volume\") pod \"payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg\" (UID: \"5b81dd7c-d225-4645-a679-53f0c1a4d6c3\") " pod="openshift-ingress/payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg" Oct 29 22:50:30.055598 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:30.055509 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t27v\" (UniqueName: \"kubernetes.io/projected/5b81dd7c-d225-4645-a679-53f0c1a4d6c3-kube-api-access-6t27v\") pod \"payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg\" (UID: \"5b81dd7c-d225-4645-a679-53f0c1a4d6c3\") " pod="openshift-ingress/payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg" Oct 29 22:50:30.156878 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:30.156843 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b81dd7c-d225-4645-a679-53f0c1a4d6c3-plugins-config-volume\") pod \"payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg\" (UID: \"5b81dd7c-d225-4645-a679-53f0c1a4d6c3\") " pod="openshift-ingress/payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg" Oct 29 22:50:30.157028 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:30.156921 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t27v\" (UniqueName: \"kubernetes.io/projected/5b81dd7c-d225-4645-a679-53f0c1a4d6c3-kube-api-access-6t27v\") pod \"payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg\" (UID: \"5b81dd7c-d225-4645-a679-53f0c1a4d6c3\") " pod="openshift-ingress/payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg" Oct 29 22:50:30.157577 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:30.157556 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b81dd7c-d225-4645-a679-53f0c1a4d6c3-plugins-config-volume\") pod \"payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg\" (UID: \"5b81dd7c-d225-4645-a679-53f0c1a4d6c3\") " pod="openshift-ingress/payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg" Oct 29 22:50:30.179689 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:30.179663 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t27v\" (UniqueName: \"kubernetes.io/projected/5b81dd7c-d225-4645-a679-53f0c1a4d6c3-kube-api-access-6t27v\") pod \"payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg\" (UID: \"5b81dd7c-d225-4645-a679-53f0c1a4d6c3\") " pod="openshift-ingress/payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg" Oct 29 22:50:30.276667 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:30.276564 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg" Oct 29 22:50:30.613994 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:30.613971 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg"] Oct 29 22:50:30.640949 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:50:30.640910 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b81dd7c_d225_4645_a679_53f0c1a4d6c3.slice/crio-72e97509a2989913dd853a57f011b971b16a3f78d15f052867d94d4a18de53cd WatchSource:0}: Error finding container 72e97509a2989913dd853a57f011b971b16a3f78d15f052867d94d4a18de53cd: Status 404 returned error can't find the container with id 72e97509a2989913dd853a57f011b971b16a3f78d15f052867d94d4a18de53cd Oct 29 22:50:30.866597 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:30.866497 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg" event={"ID":"5b81dd7c-d225-4645-a679-53f0c1a4d6c3","Type":"ContainerStarted","Data":"72e97509a2989913dd853a57f011b971b16a3f78d15f052867d94d4a18de53cd"} Oct 29 22:50:32.488952 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:32.488907 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f"] Oct 29 22:50:32.498012 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:32.497987 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" Oct 29 22:50:32.547961 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:32.547927 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f"] Oct 29 22:50:32.548289 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:32.548250 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:50:32.548407 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:32.548399 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:50:32.548473 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:32.548446 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1db2136c-serving-cert\"" Oct 29 22:50:32.548541 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:32.548526 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xrwtg\"" Oct 29 22:50:32.548596 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:32.548579 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 29 22:50:32.601560 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:32.601519 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc056f03-a438-4831-9803-eaa766186618-openshift-service-ca\") pod \"maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f\" (UID: \"fc056f03-a438-4831-9803-eaa766186618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" Oct 29 22:50:32.601727 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:32.601633 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc056f03-a438-4831-9803-eaa766186618-metrics-certs\") pod \"maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f\" (UID: \"fc056f03-a438-4831-9803-eaa766186618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" Oct 29 22:50:32.601727 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:32.601678 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc056f03-a438-4831-9803-eaa766186618-maas-api-tls\") pod \"maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f\" (UID: \"fc056f03-a438-4831-9803-eaa766186618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" Oct 29 22:50:32.601727 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:32.601715 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjq8w\" (UniqueName: \"kubernetes.io/projected/fc056f03-a438-4831-9803-eaa766186618-kube-api-access-wjq8w\") pod \"maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f\" (UID: \"fc056f03-a438-4831-9803-eaa766186618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" Oct 29 22:50:32.703865 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:32.703145 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc056f03-a438-4831-9803-eaa766186618-maas-api-tls\") pod \"maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f\" (UID: \"fc056f03-a438-4831-9803-eaa766186618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" Oct 29 22:50:32.703865 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:32.703238 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjq8w\" (UniqueName: \"kubernetes.io/projected/fc056f03-a438-4831-9803-eaa766186618-kube-api-access-wjq8w\") pod \"maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f\" (UID: \"fc056f03-a438-4831-9803-eaa766186618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" Oct 29 22:50:32.703865 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:32.703309 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc056f03-a438-4831-9803-eaa766186618-openshift-service-ca\") pod \"maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f\" (UID: \"fc056f03-a438-4831-9803-eaa766186618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" Oct 29 22:50:32.703865 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:32.703366 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc056f03-a438-4831-9803-eaa766186618-metrics-certs\") pod \"maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f\" (UID: \"fc056f03-a438-4831-9803-eaa766186618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" Oct 29 22:50:32.711398 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:32.710069 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc056f03-a438-4831-9803-eaa766186618-metrics-certs\") pod \"maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f\" (UID: \"fc056f03-a438-4831-9803-eaa766186618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" Oct 29 22:50:32.711398 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:50:32.710241 4821 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-serving-cert: secret "maas-api-e2e-ait-1db2136c-serving-cert" not found Oct 29 22:50:32.711398 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:50:32.710327 4821 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc056f03-a438-4831-9803-eaa766186618-maas-api-tls podName:fc056f03-a438-4831-9803-eaa766186618 nodeName:}" failed. No retries permitted until 2026-10-29 22:50:33.210306485 +0000 UTC m=+487.150854391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fc056f03-a438-4831-9803-eaa766186618-maas-api-tls") pod "maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" (UID: "fc056f03-a438-4831-9803-eaa766186618") : secret "maas-api-e2e-ait-1db2136c-serving-cert" not found Oct 29 22:50:32.711398 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:32.711345 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc056f03-a438-4831-9803-eaa766186618-openshift-service-ca\") pod \"maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f\" (UID: \"fc056f03-a438-4831-9803-eaa766186618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" Oct 29 22:50:32.774351 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:32.774230 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjq8w\" (UniqueName: \"kubernetes.io/projected/fc056f03-a438-4831-9803-eaa766186618-kube-api-access-wjq8w\") pod \"maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f\" (UID: \"fc056f03-a438-4831-9803-eaa766186618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" Oct 29 22:50:33.211211 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:33.211154 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc056f03-a438-4831-9803-eaa766186618-maas-api-tls\") pod \"maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f\" (UID: \"fc056f03-a438-4831-9803-eaa766186618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" Oct 29 22:50:33.213716 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:33.213691 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc056f03-a438-4831-9803-eaa766186618-maas-api-tls\") pod \"maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f\" (UID: \"fc056f03-a438-4831-9803-eaa766186618\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" Oct 29 22:50:33.414020 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:33.413745 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" Oct 29 22:50:33.757126 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:33.756875 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f"] Oct 29 22:50:33.885970 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:33.885938 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg" event={"ID":"5b81dd7c-d225-4645-a679-53f0c1a4d6c3","Type":"ContainerStarted","Data":"176265560e8fe90ef64b954ee9320e04ccdb790c56dbdfa927cc7a008066f28d"} Oct 29 22:50:33.886169 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:33.886087 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg" Oct 29 22:50:33.887164 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:33.887142 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" event={"ID":"fc056f03-a438-4831-9803-eaa766186618","Type":"ContainerStarted","Data":"d9e71ad4f44c12ab51c48704f72a8b891768f6268b9fa7979b0b9bc38fa37a4e"} Oct 29 22:50:33.942447 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:33.942176 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg" podStartSLOduration=2.028561357 podStartE2EDuration="4.94215587s" totalImagesPullingTime="2.913594513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:33.939713425 +0000 UTC m=+487.880261337" watchObservedRunningTime="2026-10-29 22:50:33.94215587 +0000 UTC m=+487.882703783" Oct 29 22:50:36.197512 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:36.197479 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9"] Oct 29 22:50:36.201426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:36.201409 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9" Oct 29 22:50:36.216193 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:36.216153 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-08e15d-serving-cert\"" Oct 29 22:50:36.222766 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:36.222739 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9"] Oct 29 22:50:36.364421 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:36.364333 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9\" (UID: \"9a799d83-b2e3-48c2-bb3a-acbffdaee0d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9" Oct 29 22:50:36.364576 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:36.364481 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sffpx\" (UniqueName: \"kubernetes.io/projected/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-kube-api-access-sffpx\") pod \"maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9\" (UID: \"9a799d83-b2e3-48c2-bb3a-acbffdaee0d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9" Oct 29 22:50:36.364616 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:36.364565 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-metrics-certs\") pod \"maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9\" (UID: \"9a799d83-b2e3-48c2-bb3a-acbffdaee0d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9" Oct 29 22:50:36.364616 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:36.364601 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-maas-api-tls\") pod \"maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9\" (UID: \"9a799d83-b2e3-48c2-bb3a-acbffdaee0d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9" Oct 29 22:50:36.466321 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:36.466285 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9\" (UID: \"9a799d83-b2e3-48c2-bb3a-acbffdaee0d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9" Oct 29 22:50:36.466524 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:36.466369 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sffpx\" (UniqueName: \"kubernetes.io/projected/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-kube-api-access-sffpx\") pod \"maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9\" (UID: \"9a799d83-b2e3-48c2-bb3a-acbffdaee0d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9" Oct 29 22:50:36.466524 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:36.466413 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-metrics-certs\") pod \"maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9\" (UID: \"9a799d83-b2e3-48c2-bb3a-acbffdaee0d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9" Oct 29 22:50:36.466524 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:36.466442 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-maas-api-tls\") pod \"maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9\" (UID: \"9a799d83-b2e3-48c2-bb3a-acbffdaee0d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9" Oct 29 22:50:36.468531 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:36.467910 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9\" (UID: \"9a799d83-b2e3-48c2-bb3a-acbffdaee0d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9" Oct 29 22:50:36.469855 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:36.469825 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-maas-api-tls\") pod \"maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9\" (UID: \"9a799d83-b2e3-48c2-bb3a-acbffdaee0d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9" Oct 29 22:50:36.471267 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:36.471242 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-metrics-certs\") pod \"maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9\" (UID: \"9a799d83-b2e3-48c2-bb3a-acbffdaee0d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9" Oct 29 22:50:36.489599 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:36.489569 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sffpx\" (UniqueName: \"kubernetes.io/projected/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-kube-api-access-sffpx\") pod \"maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9\" (UID: \"9a799d83-b2e3-48c2-bb3a-acbffdaee0d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9" Oct 29 22:50:36.514442 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:36.514414 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9" Oct 29 22:50:36.819138 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:36.819111 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9"] Oct 29 22:50:36.821772 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:50:36.821747 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a799d83_b2e3_48c2_bb3a_acbffdaee0d2.slice/crio-ff49aa8abb6f27ec7025db45db2fa49961b4e7ed70b2ba52c0a44d817b259b62 WatchSource:0}: Error finding container ff49aa8abb6f27ec7025db45db2fa49961b4e7ed70b2ba52c0a44d817b259b62: Status 404 returned error can't find the container with id ff49aa8abb6f27ec7025db45db2fa49961b4e7ed70b2ba52c0a44d817b259b62 Oct 29 22:50:36.907425 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:36.907393 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" event={"ID":"fc056f03-a438-4831-9803-eaa766186618","Type":"ContainerStarted","Data":"f48190e6defe90ac742b09dd021f80e5ad3913c06da67fc7706d49cb12acfce3"} Oct 29 22:50:36.907636 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:36.907481 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" Oct 29 22:50:36.908881 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:36.908852 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9" event={"ID":"9a799d83-b2e3-48c2-bb3a-acbffdaee0d2","Type":"ContainerStarted","Data":"ff49aa8abb6f27ec7025db45db2fa49961b4e7ed70b2ba52c0a44d817b259b62"} Oct 29 22:50:36.952311 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:36.952251 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" podStartSLOduration=2.604469114 podStartE2EDuration="4.952232167s" totalImagesPullingTime="2.347763053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:36.951430483 +0000 UTC m=+490.891978396" watchObservedRunningTime="2026-10-29 22:50:36.952232167 +0000 UTC m=+490.892780081" Oct 29 22:50:37.915984 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:37.915949 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9" event={"ID":"9a799d83-b2e3-48c2-bb3a-acbffdaee0d2","Type":"ContainerStarted","Data":"711055e310e0f6037c0dea470743fbe9094df1dfc0685991e80255ced76a2bbc"} Oct 29 22:50:37.916539 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:37.916101 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9" Oct 29 22:50:37.917872 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:37.917846 4821 generic.go:356] "Generic (PLEG): container finished" podID="61db0858-830e-4190-8d17-6b0a41497b42" containerID="71c6e470859fdc24c85a59997ddb8be30488cc9d3466bc43c00751258366dfd6" exitCode=0 Oct 29 22:50:37.918004 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:37.917924 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" event={"ID":"61db0858-830e-4190-8d17-6b0a41497b42","Type":"ContainerDied","Data":"71c6e470859fdc24c85a59997ddb8be30488cc9d3466bc43c00751258366dfd6"} Oct 29 22:50:37.918474 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:37.918456 4821 scope.go:122] "RemoveContainer" containerID="71c6e470859fdc24c85a59997ddb8be30488cc9d3466bc43c00751258366dfd6" Oct 29 22:50:37.919829 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:37.919793 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:50:37.919932 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:37.919884 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:50:37.920003 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:37.919930 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:50:37.966115 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:37.966069 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9" podStartSLOduration=1.629145581 podStartE2EDuration="1.966055223s" totalImagesPullingTime="336.909642ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:37.963654516 +0000 UTC m=+491.904202429" watchObservedRunningTime="2026-10-29 22:50:37.966055223 +0000 UTC m=+491.906603132" Oct 29 22:50:38.801256 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:38.798289 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:38.947437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:38.947367 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" event={"ID":"61db0858-830e-4190-8d17-6b0a41497b42","Type":"ContainerStarted","Data":"37e96e82934f85ef176140ac33967ca3e355a6ee3ce88d192d3872ca7b8dee3f"} Oct 29 22:50:39.550450 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:39.550415 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:39.551637 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:39.551611 4821 patch_prober.go:29] interesting pod/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 29 22:50:39.551760 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:39.551669 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" podUID="61db0858-830e-4190-8d17-6b0a41497b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 29 22:50:40.550981 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:40.550948 4821 patch_prober.go:29] interesting pod/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 29 22:50:40.551450 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:40.551006 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" podUID="61db0858-830e-4190-8d17-6b0a41497b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 29 22:50:41.550365 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:41.550329 4821 patch_prober.go:29] interesting pod/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 29 22:50:41.550553 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:41.550385 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" podUID="61db0858-830e-4190-8d17-6b0a41497b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 29 22:50:42.551222 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:42.551015 4821 patch_prober.go:29] interesting pod/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 29 22:50:42.551222 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:42.551073 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" podUID="61db0858-830e-4190-8d17-6b0a41497b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 29 22:50:42.947433 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:42.947395 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" Oct 29 22:50:43.550733 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:43.550691 4821 patch_prober.go:29] interesting pod/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 29 22:50:43.550940 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:43.550768 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" podUID="61db0858-830e-4190-8d17-6b0a41497b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 29 22:50:43.953667 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:43.953638 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9" Oct 29 22:50:44.555818 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:44.555786 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:44.556192 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:44.556165 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:44.557129 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:44.557110 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:50:44.897331 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:44.897306 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg" Oct 29 22:50:45.729623 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.729590 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk"] Oct 29 22:50:45.735741 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.735712 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:50:45.755171 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.755147 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk"] Oct 29 22:50:45.764168 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.764141 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-31226c\"/\"openshift-service-ca.crt\"" Oct 29 22:50:45.764361 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.764251 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-31226c\"/\"kube-root-ca.crt\"" Oct 29 22:50:45.764361 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.764249 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-31226c\"/\"simulator-w6-31226c-kserve-self-signed-certs\"" Oct 29 22:50:45.764361 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.764320 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-31226c\"/\"default-dockercfg-hqr2k\"" Oct 29 22:50:45.780781 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.780741 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-home\") pod \"simulator-w6-31226c-kserve-78ddf98fb8-2fmxk\" (UID: \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\") " pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:50:45.780781 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.780786 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-tls-certs\") pod \"simulator-w6-31226c-kserve-78ddf98fb8-2fmxk\" (UID: \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\") " pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:50:45.780978 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.780818 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-dshm\") pod \"simulator-w6-31226c-kserve-78ddf98fb8-2fmxk\" (UID: \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\") " pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:50:45.780978 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.780939 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-model-cache\") pod \"simulator-w6-31226c-kserve-78ddf98fb8-2fmxk\" (UID: \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\") " pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:50:45.781058 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.780985 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-tmp-dir\") pod \"simulator-w6-31226c-kserve-78ddf98fb8-2fmxk\" (UID: \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\") " pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:50:45.781093 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.781065 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcf6b\" (UniqueName: \"kubernetes.io/projected/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-kube-api-access-xcf6b\") pod \"simulator-w6-31226c-kserve-78ddf98fb8-2fmxk\" (UID: \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\") " pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:50:45.881563 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.881521 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-home\") pod \"simulator-w6-31226c-kserve-78ddf98fb8-2fmxk\" (UID: \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\") " pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:50:45.881769 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.881570 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-tls-certs\") pod \"simulator-w6-31226c-kserve-78ddf98fb8-2fmxk\" (UID: \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\") " pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:50:45.881769 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.881596 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-dshm\") pod \"simulator-w6-31226c-kserve-78ddf98fb8-2fmxk\" (UID: \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\") " pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:50:45.881769 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.881637 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-model-cache\") pod \"simulator-w6-31226c-kserve-78ddf98fb8-2fmxk\" (UID: \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\") " pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:50:45.881769 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.881677 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-tmp-dir\") pod \"simulator-w6-31226c-kserve-78ddf98fb8-2fmxk\" (UID: \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\") " pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:50:45.881769 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.881718 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcf6b\" (UniqueName: \"kubernetes.io/projected/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-kube-api-access-xcf6b\") pod \"simulator-w6-31226c-kserve-78ddf98fb8-2fmxk\" (UID: \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\") " pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:50:45.882028 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.881957 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-home\") pod \"simulator-w6-31226c-kserve-78ddf98fb8-2fmxk\" (UID: \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\") " pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:50:45.882083 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.882021 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-model-cache\") pod \"simulator-w6-31226c-kserve-78ddf98fb8-2fmxk\" (UID: \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\") " pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:50:45.882224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.882159 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-tmp-dir\") pod \"simulator-w6-31226c-kserve-78ddf98fb8-2fmxk\" (UID: \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\") " pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:50:45.885110 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.884870 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-dshm\") pod \"simulator-w6-31226c-kserve-78ddf98fb8-2fmxk\" (UID: \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\") " pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:50:45.885247 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.885166 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-tls-certs\") pod \"simulator-w6-31226c-kserve-78ddf98fb8-2fmxk\" (UID: \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\") " pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:50:45.905093 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:45.905060 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcf6b\" (UniqueName: \"kubernetes.io/projected/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-kube-api-access-xcf6b\") pod \"simulator-w6-31226c-kserve-78ddf98fb8-2fmxk\" (UID: \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\") " pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:50:46.050445 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:46.050351 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:50:46.351545 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:46.351520 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk"] Oct 29 22:50:46.353435 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:50:46.353412 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa4898b3_8ad1_4d3a_a516_9f2a6d3468b8.slice/crio-c652939119a70307000355db063d8a9c41ecbb595b4c6ac0c41c4df3b6366f18 WatchSource:0}: Error finding container c652939119a70307000355db063d8a9c41ecbb595b4c6ac0c41c4df3b6366f18: Status 404 returned error can't find the container with id c652939119a70307000355db063d8a9c41ecbb595b4c6ac0c41c4df3b6366f18 Oct 29 22:50:46.992578 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:46.992525 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" event={"ID":"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8","Type":"ContainerStarted","Data":"c652939119a70307000355db063d8a9c41ecbb595b4c6ac0c41c4df3b6366f18"} Oct 29 22:50:47.999659 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:47.999615 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" event={"ID":"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8","Type":"ContainerStarted","Data":"c1b6b0bdbb4fda69b750bbdc6e3b68c5e403eb266f93c05674c6a3d86fefe5a9"} Oct 29 22:50:48.044982 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:48.044127 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podStartSLOduration=1.9055540340000001 podStartE2EDuration="3.044105015s" totalImagesPullingTime="1.138550981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:48.042576507 +0000 UTC m=+501.983124414" watchObservedRunningTime="2026-10-29 22:50:48.044105015 +0000 UTC m=+501.984652967" Oct 29 22:50:48.051386 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:48.051353 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:50:53.885312 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:50:53.885260 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:50:56.050947 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:56.050909 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:50:56.063654 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:56.063627 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:50:57.056245 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:57.056217 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:50:59.622781 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.622749 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6"] Oct 29 22:50:59.627244 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.627185 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:50:59.653842 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.653810 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6"] Oct 29 22:50:59.670628 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.670591 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-08e15d\"/\"kube-root-ca.crt\"" Oct 29 22:50:59.670775 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.670627 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-08e15d\"/\"default-dockercfg-246bt\"" Oct 29 22:50:59.670775 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.670674 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-08e15d\"/\"openshift-service-ca.crt\"" Oct 29 22:50:59.671941 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.671926 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-08e15d\"/\"simulator-w0-08e15d-kserve-self-signed-certs\"" Oct 29 22:50:59.713091 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.713049 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-tmp-dir\") pod \"simulator-w0-08e15d-kserve-7d75cc98bb-brwp6\" (UID: \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\") " pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:50:59.713276 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.713118 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-home\") pod \"simulator-w0-08e15d-kserve-7d75cc98bb-brwp6\" (UID: \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\") " pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:50:59.713276 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.713184 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-dshm\") pod \"simulator-w0-08e15d-kserve-7d75cc98bb-brwp6\" (UID: \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\") " pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:50:59.713356 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.713272 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeacdd0d-a01d-498c-bf12-f384dc79d57f-tls-certs\") pod \"simulator-w0-08e15d-kserve-7d75cc98bb-brwp6\" (UID: \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\") " pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:50:59.713391 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.713353 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-model-cache\") pod \"simulator-w0-08e15d-kserve-7d75cc98bb-brwp6\" (UID: \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\") " pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:50:59.713436 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.713411 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v89r2\" (UniqueName: \"kubernetes.io/projected/eeacdd0d-a01d-498c-bf12-f384dc79d57f-kube-api-access-v89r2\") pod \"simulator-w0-08e15d-kserve-7d75cc98bb-brwp6\" (UID: \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\") " pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:50:59.814822 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.814791 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89r2\" (UniqueName: \"kubernetes.io/projected/eeacdd0d-a01d-498c-bf12-f384dc79d57f-kube-api-access-v89r2\") pod \"simulator-w0-08e15d-kserve-7d75cc98bb-brwp6\" (UID: \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\") " pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:50:59.815002 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.814850 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-tmp-dir\") pod \"simulator-w0-08e15d-kserve-7d75cc98bb-brwp6\" (UID: \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\") " pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:50:59.815002 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.814885 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-home\") pod \"simulator-w0-08e15d-kserve-7d75cc98bb-brwp6\" (UID: \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\") " pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:50:59.815002 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.814903 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-dshm\") pod \"simulator-w0-08e15d-kserve-7d75cc98bb-brwp6\" (UID: \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\") " pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:50:59.815002 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.814921 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeacdd0d-a01d-498c-bf12-f384dc79d57f-tls-certs\") pod \"simulator-w0-08e15d-kserve-7d75cc98bb-brwp6\" (UID: \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\") " pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:50:59.815002 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.814957 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-model-cache\") pod \"simulator-w0-08e15d-kserve-7d75cc98bb-brwp6\" (UID: \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\") " pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:50:59.815359 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.815333 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-home\") pod \"simulator-w0-08e15d-kserve-7d75cc98bb-brwp6\" (UID: \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\") " pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:50:59.815439 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.815375 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-tmp-dir\") pod \"simulator-w0-08e15d-kserve-7d75cc98bb-brwp6\" (UID: \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\") " pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:50:59.815439 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.815433 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-model-cache\") pod \"simulator-w0-08e15d-kserve-7d75cc98bb-brwp6\" (UID: \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\") " pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:50:59.817459 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.817433 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-dshm\") pod \"simulator-w0-08e15d-kserve-7d75cc98bb-brwp6\" (UID: \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\") " pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:50:59.817907 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.817886 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeacdd0d-a01d-498c-bf12-f384dc79d57f-tls-certs\") pod \"simulator-w0-08e15d-kserve-7d75cc98bb-brwp6\" (UID: \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\") " pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:50:59.838077 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.838046 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v89r2\" (UniqueName: \"kubernetes.io/projected/eeacdd0d-a01d-498c-bf12-f384dc79d57f-kube-api-access-v89r2\") pod \"simulator-w0-08e15d-kserve-7d75cc98bb-brwp6\" (UID: \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\") " pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:50:59.940925 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:50:59.940838 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:51:00.243591 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:00.243567 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6"] Oct 29 22:51:00.245976 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:51:00.245945 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeeacdd0d_a01d_498c_bf12_f384dc79d57f.slice/crio-be38575c459b81d015ad3849406a825318e3f1b73787c648071aa77c090f7734 WatchSource:0}: Error finding container be38575c459b81d015ad3849406a825318e3f1b73787c648071aa77c090f7734: Status 404 returned error can't find the container with id be38575c459b81d015ad3849406a825318e3f1b73787c648071aa77c090f7734 Oct 29 22:51:01.071725 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:01.071682 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" event={"ID":"eeacdd0d-a01d-498c-bf12-f384dc79d57f","Type":"ContainerStarted","Data":"321f062041e5c67688472c8b1eaa42514a1862591aa1b37d389ea82c74470f21"} Oct 29 22:51:01.072284 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:01.071737 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" event={"ID":"eeacdd0d-a01d-498c-bf12-f384dc79d57f","Type":"ContainerStarted","Data":"be38575c459b81d015ad3849406a825318e3f1b73787c648071aa77c090f7734"} Oct 29 22:51:01.116315 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:01.116266 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podStartSLOduration=2.116248972 podStartE2EDuration="2.116248972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:01.114472526 +0000 UTC m=+515.055020439" watchObservedRunningTime="2026-10-29 22:51:01.116248972 +0000 UTC m=+515.056796881" Oct 29 22:51:01.941223 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:01.941166 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:51:05.976003 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:05.975937 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg"] Oct 29 22:51:07.028491 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:07.028451 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2"] Oct 29 22:51:07.028979 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:07.028721 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" podUID="61db0858-830e-4190-8d17-6b0a41497b42" containerName="istio-proxy" containerID="cri-o://37e96e82934f85ef176140ac33967ca3e355a6ee3ce88d192d3872ca7b8dee3f" gracePeriod=30 Oct 29 22:51:09.941529 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:09.941489 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:51:09.954922 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:09.954896 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:51:10.131296 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:10.131256 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:51:10.977585 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:10.977530 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg" podUID="5b81dd7c-d225-4645-a679-53f0c1a4d6c3" containerName="payload-processing" containerID="cri-o://176265560e8fe90ef64b954ee9320e04ccdb790c56dbdfa927cc7a008066f28d" gracePeriod=25 Oct 29 22:51:11.128367 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:11.128335 4821 generic.go:356] "Generic (PLEG): container finished" podID="5b81dd7c-d225-4645-a679-53f0c1a4d6c3" containerID="176265560e8fe90ef64b954ee9320e04ccdb790c56dbdfa927cc7a008066f28d" exitCode=0 Oct 29 22:51:11.128970 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:11.128951 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg" event={"ID":"5b81dd7c-d225-4645-a679-53f0c1a4d6c3","Type":"ContainerDied","Data":"176265560e8fe90ef64b954ee9320e04ccdb790c56dbdfa927cc7a008066f28d"} Oct 29 22:51:11.246429 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:11.246406 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg" Oct 29 22:51:11.342097 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:11.342059 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b81dd7c-d225-4645-a679-53f0c1a4d6c3-kube-api-access-6t27v\" (UniqueName: \"kubernetes.io/projected/5b81dd7c-d225-4645-a679-53f0c1a4d6c3-kube-api-access-6t27v\") pod \"5b81dd7c-d225-4645-a679-53f0c1a4d6c3\" (UID: \"5b81dd7c-d225-4645-a679-53f0c1a4d6c3\") " Oct 29 22:51:11.342299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:11.342255 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b81dd7c-d225-4645-a679-53f0c1a4d6c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b81dd7c-d225-4645-a679-53f0c1a4d6c3-plugins-config-volume\") pod \"5b81dd7c-d225-4645-a679-53f0c1a4d6c3\" (UID: \"5b81dd7c-d225-4645-a679-53f0c1a4d6c3\") " Oct 29 22:51:11.342720 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:11.342688 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b81dd7c-d225-4645-a679-53f0c1a4d6c3-plugins-config-volume" pod "5b81dd7c-d225-4645-a679-53f0c1a4d6c3" (UID: "5b81dd7c-d225-4645-a679-53f0c1a4d6c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:11.344667 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:11.344640 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b81dd7c-d225-4645-a679-53f0c1a4d6c3-kube-api-access-6t27v" pod "5b81dd7c-d225-4645-a679-53f0c1a4d6c3" (UID: "5b81dd7c-d225-4645-a679-53f0c1a4d6c3"). InnerVolumeSpecName "kube-api-access-6t27v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:11.443651 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:11.443619 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t27v\" (UniqueName: \"kubernetes.io/projected/5b81dd7c-d225-4645-a679-53f0c1a4d6c3-kube-api-access-6t27v\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:51:11.443651 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:11.443645 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b81dd7c-d225-4645-a679-53f0c1a4d6c3-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:51:11.936172 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:11.936134 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt"] Oct 29 22:51:11.936676 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:11.936660 4821 cpu_manager.go:414] "containerMap: removing container" podUID="5b81dd7c-d225-4645-a679-53f0c1a4d6c3" containerName="payload-processing" Oct 29 22:51:11.936720 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:11.936679 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b81dd7c-d225-4645-a679-53f0c1a4d6c3" containerName="payload-processing" Oct 29 22:51:11.936786 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:11.936775 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b81dd7c-d225-4645-a679-53f0c1a4d6c3" containerName="payload-processing" Oct 29 22:51:11.940136 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:11.940118 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt" Oct 29 22:51:11.959484 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:11.959461 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-063b46\"" Oct 29 22:51:11.962293 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:11.962268 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt"] Oct 29 22:51:11.968121 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:11.968091 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-063b46-dockercfg-b49j7\"" Oct 29 22:51:12.039810 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.038719 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz"] Oct 29 22:51:12.052999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.052965 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsqcf\" (UniqueName: \"kubernetes.io/projected/9a821056-9bb0-4430-ab80-d6c44d3fa4f5-kube-api-access-lsqcf\") pod \"payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt\" (UID: \"9a821056-9bb0-4430-ab80-d6c44d3fa4f5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt" Oct 29 22:51:12.053135 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.053034 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a821056-9bb0-4430-ab80-d6c44d3fa4f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt\" (UID: \"9a821056-9bb0-4430-ab80-d6c44d3fa4f5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt" Oct 29 22:51:12.056261 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.056235 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz" Oct 29 22:51:12.065046 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.065020 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz"] Oct 29 22:51:12.137531 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.137501 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg" Oct 29 22:51:12.137679 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.137530 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg" event={"ID":"5b81dd7c-d225-4645-a679-53f0c1a4d6c3","Type":"ContainerDied","Data":"72e97509a2989913dd853a57f011b971b16a3f78d15f052867d94d4a18de53cd"} Oct 29 22:51:12.137679 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.137599 4821 scope.go:122] "RemoveContainer" containerID="176265560e8fe90ef64b954ee9320e04ccdb790c56dbdfa927cc7a008066f28d" Oct 29 22:51:12.154512 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.154483 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvbdd\" (UniqueName: \"kubernetes.io/projected/9809020b-a844-44fa-86af-6565434de12d-kube-api-access-lvbdd\") pod \"payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz\" (UID: \"9809020b-a844-44fa-86af-6565434de12d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz" Oct 29 22:51:12.154606 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.154574 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9809020b-a844-44fa-86af-6565434de12d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz\" (UID: \"9809020b-a844-44fa-86af-6565434de12d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz" Oct 29 22:51:12.154712 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.154690 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsqcf\" (UniqueName: \"kubernetes.io/projected/9a821056-9bb0-4430-ab80-d6c44d3fa4f5-kube-api-access-lsqcf\") pod \"payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt\" (UID: \"9a821056-9bb0-4430-ab80-d6c44d3fa4f5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt" Oct 29 22:51:12.154775 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.154759 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a821056-9bb0-4430-ab80-d6c44d3fa4f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt\" (UID: \"9a821056-9bb0-4430-ab80-d6c44d3fa4f5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt" Oct 29 22:51:12.155577 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.155558 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a821056-9bb0-4430-ab80-d6c44d3fa4f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt\" (UID: \"9a821056-9bb0-4430-ab80-d6c44d3fa4f5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt" Oct 29 22:51:12.177237 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.177194 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsqcf\" (UniqueName: \"kubernetes.io/projected/9a821056-9bb0-4430-ab80-d6c44d3fa4f5-kube-api-access-lsqcf\") pod \"payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt\" (UID: \"9a821056-9bb0-4430-ab80-d6c44d3fa4f5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt" Oct 29 22:51:12.186333 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.186271 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg"] Oct 29 22:51:12.191934 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.191911 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1db2136c-5d9b57f89d-fc8zg"] Oct 29 22:51:12.254320 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.254288 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt" Oct 29 22:51:12.255703 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.255677 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvbdd\" (UniqueName: \"kubernetes.io/projected/9809020b-a844-44fa-86af-6565434de12d-kube-api-access-lvbdd\") pod \"payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz\" (UID: \"9809020b-a844-44fa-86af-6565434de12d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz" Oct 29 22:51:12.255844 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.255734 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9809020b-a844-44fa-86af-6565434de12d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz\" (UID: \"9809020b-a844-44fa-86af-6565434de12d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz" Oct 29 22:51:12.256581 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.256561 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9809020b-a844-44fa-86af-6565434de12d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz\" (UID: \"9809020b-a844-44fa-86af-6565434de12d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz" Oct 29 22:51:12.289105 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.286745 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvbdd\" (UniqueName: \"kubernetes.io/projected/9809020b-a844-44fa-86af-6565434de12d-kube-api-access-lvbdd\") pod \"payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz\" (UID: \"9809020b-a844-44fa-86af-6565434de12d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz" Oct 29 22:51:12.291105 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.291079 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q"] Oct 29 22:51:12.295902 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.295869 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:51:12.302786 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.302381 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:51:12.315669 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.315624 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-31226c\"/\"premium-w6-31226c-kserve-self-signed-certs\"" Oct 29 22:51:12.322531 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.322502 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q"] Oct 29 22:51:12.438257 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.438157 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz" Oct 29 22:51:12.458171 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.458132 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61db0858-830e-4190-8d17-6b0a41497b42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61db0858-830e-4190-8d17-6b0a41497b42-istiod-ca-cert\") pod \"61db0858-830e-4190-8d17-6b0a41497b42\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " Oct 29 22:51:12.458329 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.458189 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-workload-certs\") pod \"61db0858-830e-4190-8d17-6b0a41497b42\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " Oct 29 22:51:12.458329 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.458247 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-istio-data\") pod \"61db0858-830e-4190-8d17-6b0a41497b42\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " Oct 29 22:51:12.458329 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.458273 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61db0858-830e-4190-8d17-6b0a41497b42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61db0858-830e-4190-8d17-6b0a41497b42-istio-podinfo\") pod \"61db0858-830e-4190-8d17-6b0a41497b42\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " Oct 29 22:51:12.458329 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.458301 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-credential-socket\") pod \"61db0858-830e-4190-8d17-6b0a41497b42\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " Oct 29 22:51:12.458329 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.458321 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-workload-socket\") pod \"61db0858-830e-4190-8d17-6b0a41497b42\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " Oct 29 22:51:12.458596 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.458359 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61db0858-830e-4190-8d17-6b0a41497b42-istio-token\" (UniqueName: \"kubernetes.io/projected/61db0858-830e-4190-8d17-6b0a41497b42-istio-token\") pod \"61db0858-830e-4190-8d17-6b0a41497b42\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " Oct 29 22:51:12.458596 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.458414 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-istio-envoy\") pod \"61db0858-830e-4190-8d17-6b0a41497b42\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " Oct 29 22:51:12.458596 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.458430 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-workload-certs" pod "61db0858-830e-4190-8d17-6b0a41497b42" (UID: "61db0858-830e-4190-8d17-6b0a41497b42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:12.458596 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.458443 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61db0858-830e-4190-8d17-6b0a41497b42-kube-api-access-xkw9s\" (UniqueName: \"kubernetes.io/projected/61db0858-830e-4190-8d17-6b0a41497b42-kube-api-access-xkw9s\") pod \"61db0858-830e-4190-8d17-6b0a41497b42\" (UID: \"61db0858-830e-4190-8d17-6b0a41497b42\") " Oct 29 22:51:12.458596 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.458511 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61db0858-830e-4190-8d17-6b0a41497b42-istiod-ca-cert" pod "61db0858-830e-4190-8d17-6b0a41497b42" (UID: "61db0858-830e-4190-8d17-6b0a41497b42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:12.458851 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.458690 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npz4n\" (UniqueName: \"kubernetes.io/projected/62a152d7-e218-4f71-b9d4-265acdf8c16e-kube-api-access-npz4n\") pod \"premium-w6-31226c-kserve-76b64b7dfc-qld2q\" (UID: \"62a152d7-e218-4f71-b9d4-265acdf8c16e\") " pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:51:12.458851 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.458747 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-dshm\") pod \"premium-w6-31226c-kserve-76b64b7dfc-qld2q\" (UID: \"62a152d7-e218-4f71-b9d4-265acdf8c16e\") " pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:51:12.458851 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.458798 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-home\") pod \"premium-w6-31226c-kserve-76b64b7dfc-qld2q\" (UID: \"62a152d7-e218-4f71-b9d4-265acdf8c16e\") " pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:51:12.458851 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.458830 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a152d7-e218-4f71-b9d4-265acdf8c16e-tls-certs\") pod \"premium-w6-31226c-kserve-76b64b7dfc-qld2q\" (UID: \"62a152d7-e218-4f71-b9d4-265acdf8c16e\") " pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:51:12.459059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.458916 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-tmp-dir\") pod \"premium-w6-31226c-kserve-76b64b7dfc-qld2q\" (UID: \"62a152d7-e218-4f71-b9d4-265acdf8c16e\") " pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:51:12.459059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.458981 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-model-cache\") pod \"premium-w6-31226c-kserve-76b64b7dfc-qld2q\" (UID: \"62a152d7-e218-4f71-b9d4-265acdf8c16e\") " pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:51:12.459059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.459033 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61db0858-830e-4190-8d17-6b0a41497b42-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:51:12.459059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.459050 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:51:12.459280 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.458749 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-credential-socket" pod "61db0858-830e-4190-8d17-6b0a41497b42" (UID: "61db0858-830e-4190-8d17-6b0a41497b42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:12.459280 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.458909 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-istio-data" pod "61db0858-830e-4190-8d17-6b0a41497b42" (UID: "61db0858-830e-4190-8d17-6b0a41497b42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:12.459863 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.459835 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-workload-socket" pod "61db0858-830e-4190-8d17-6b0a41497b42" (UID: "61db0858-830e-4190-8d17-6b0a41497b42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:12.461321 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.461292 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61db0858-830e-4190-8d17-6b0a41497b42-kube-api-access-xkw9s" pod "61db0858-830e-4190-8d17-6b0a41497b42" (UID: "61db0858-830e-4190-8d17-6b0a41497b42"). InnerVolumeSpecName "kube-api-access-xkw9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:12.461653 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.461630 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61db0858-830e-4190-8d17-6b0a41497b42-istio-podinfo" pod "61db0858-830e-4190-8d17-6b0a41497b42" (UID: "61db0858-830e-4190-8d17-6b0a41497b42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:51:12.461740 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.461685 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61db0858-830e-4190-8d17-6b0a41497b42-istio-token" pod "61db0858-830e-4190-8d17-6b0a41497b42" (UID: "61db0858-830e-4190-8d17-6b0a41497b42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:12.461908 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.461894 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-istio-envoy" pod "61db0858-830e-4190-8d17-6b0a41497b42" (UID: "61db0858-830e-4190-8d17-6b0a41497b42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:12.559990 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.559950 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npz4n\" (UniqueName: \"kubernetes.io/projected/62a152d7-e218-4f71-b9d4-265acdf8c16e-kube-api-access-npz4n\") pod \"premium-w6-31226c-kserve-76b64b7dfc-qld2q\" (UID: \"62a152d7-e218-4f71-b9d4-265acdf8c16e\") " pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:51:12.560169 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.560018 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-dshm\") pod \"premium-w6-31226c-kserve-76b64b7dfc-qld2q\" (UID: \"62a152d7-e218-4f71-b9d4-265acdf8c16e\") " pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:51:12.560169 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.560060 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-home\") pod \"premium-w6-31226c-kserve-76b64b7dfc-qld2q\" (UID: \"62a152d7-e218-4f71-b9d4-265acdf8c16e\") " pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:51:12.560169 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.560087 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a152d7-e218-4f71-b9d4-265acdf8c16e-tls-certs\") pod \"premium-w6-31226c-kserve-76b64b7dfc-qld2q\" (UID: \"62a152d7-e218-4f71-b9d4-265acdf8c16e\") " pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:51:12.560169 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.560157 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-tmp-dir\") pod \"premium-w6-31226c-kserve-76b64b7dfc-qld2q\" (UID: \"62a152d7-e218-4f71-b9d4-265acdf8c16e\") " pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:51:12.560390 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.560234 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-model-cache\") pod \"premium-w6-31226c-kserve-76b64b7dfc-qld2q\" (UID: \"62a152d7-e218-4f71-b9d4-265acdf8c16e\") " pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:51:12.560390 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.560295 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61db0858-830e-4190-8d17-6b0a41497b42-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:51:12.560390 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.560312 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:51:12.560390 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.560326 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkw9s\" (UniqueName: \"kubernetes.io/projected/61db0858-830e-4190-8d17-6b0a41497b42-kube-api-access-xkw9s\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:51:12.560390 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.560340 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:51:12.560390 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.560353 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61db0858-830e-4190-8d17-6b0a41497b42-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:51:12.560390 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.560365 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:51:12.560696 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.560379 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61db0858-830e-4190-8d17-6b0a41497b42-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:51:12.560696 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.560576 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-home\") pod \"premium-w6-31226c-kserve-76b64b7dfc-qld2q\" (UID: \"62a152d7-e218-4f71-b9d4-265acdf8c16e\") " pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:51:12.560696 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.560661 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-tmp-dir\") pod \"premium-w6-31226c-kserve-76b64b7dfc-qld2q\" (UID: \"62a152d7-e218-4f71-b9d4-265acdf8c16e\") " pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:51:12.560696 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.560666 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-model-cache\") pod \"premium-w6-31226c-kserve-76b64b7dfc-qld2q\" (UID: \"62a152d7-e218-4f71-b9d4-265acdf8c16e\") " pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:51:12.562413 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.562388 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-dshm\") pod \"premium-w6-31226c-kserve-76b64b7dfc-qld2q\" (UID: \"62a152d7-e218-4f71-b9d4-265acdf8c16e\") " pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:51:12.563093 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.563076 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a152d7-e218-4f71-b9d4-265acdf8c16e-tls-certs\") pod \"premium-w6-31226c-kserve-76b64b7dfc-qld2q\" (UID: \"62a152d7-e218-4f71-b9d4-265acdf8c16e\") " pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:51:12.573378 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.573357 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt"] Oct 29 22:51:12.575047 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:51:12.575013 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a821056_9bb0_4430_ab80_d6c44d3fa4f5.slice/crio-56c007ae28bff4a54199eaf083449dc9bd7af1f22cc23e3aac38604750584136 WatchSource:0}: Error finding container 56c007ae28bff4a54199eaf083449dc9bd7af1f22cc23e3aac38604750584136: Status 404 returned error can't find the container with id 56c007ae28bff4a54199eaf083449dc9bd7af1f22cc23e3aac38604750584136 Oct 29 22:51:12.582673 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.582650 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npz4n\" (UniqueName: \"kubernetes.io/projected/62a152d7-e218-4f71-b9d4-265acdf8c16e-kube-api-access-npz4n\") pod \"premium-w6-31226c-kserve-76b64b7dfc-qld2q\" (UID: \"62a152d7-e218-4f71-b9d4-265acdf8c16e\") " pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:51:12.623700 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.623676 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:51:12.755348 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.755318 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz"] Oct 29 22:51:12.756824 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:51:12.756791 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9809020b_a844_44fa_86af_6565434de12d.slice/crio-06013545c61dcef6ebaf9ff520ccaba551305e13622c10172235908b1d0dd011 WatchSource:0}: Error finding container 06013545c61dcef6ebaf9ff520ccaba551305e13622c10172235908b1d0dd011: Status 404 returned error can't find the container with id 06013545c61dcef6ebaf9ff520ccaba551305e13622c10172235908b1d0dd011 Oct 29 22:51:12.924604 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:12.924055 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q"] Oct 29 22:51:12.926120 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:51:12.926096 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62a152d7_e218_4f71_b9d4_265acdf8c16e.slice/crio-835994da901ae53002154c84551f42b95cc46835eab6ace65cc9617d64d8be98 WatchSource:0}: Error finding container 835994da901ae53002154c84551f42b95cc46835eab6ace65cc9617d64d8be98: Status 404 returned error can't find the container with id 835994da901ae53002154c84551f42b95cc46835eab6ace65cc9617d64d8be98 Oct 29 22:51:13.144269 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:13.144236 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt" event={"ID":"9a821056-9bb0-4430-ab80-d6c44d3fa4f5","Type":"ContainerStarted","Data":"fe986ccc7ea53813b6f95d494d5f8b6eb50088ea9e4378031b7cb0ce83db9e06"} Oct 29 22:51:13.144685 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:13.144291 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt" event={"ID":"9a821056-9bb0-4430-ab80-d6c44d3fa4f5","Type":"ContainerStarted","Data":"56c007ae28bff4a54199eaf083449dc9bd7af1f22cc23e3aac38604750584136"} Oct 29 22:51:13.144685 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:13.144349 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt" Oct 29 22:51:13.146043 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:13.146019 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" event={"ID":"62a152d7-e218-4f71-b9d4-265acdf8c16e","Type":"ContainerStarted","Data":"d3e48001cec67eafc9eb81d2d612433f1360b024ae494a4b5a306d20ca4991fe"} Oct 29 22:51:13.146140 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:13.146050 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" event={"ID":"62a152d7-e218-4f71-b9d4-265acdf8c16e","Type":"ContainerStarted","Data":"835994da901ae53002154c84551f42b95cc46835eab6ace65cc9617d64d8be98"} Oct 29 22:51:13.147666 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:13.147642 4821 generic.go:356] "Generic (PLEG): container finished" podID="61db0858-830e-4190-8d17-6b0a41497b42" containerID="37e96e82934f85ef176140ac33967ca3e355a6ee3ce88d192d3872ca7b8dee3f" exitCode=0 Oct 29 22:51:13.147739 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:13.147707 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" Oct 29 22:51:13.147739 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:13.147721 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" event={"ID":"61db0858-830e-4190-8d17-6b0a41497b42","Type":"ContainerDied","Data":"37e96e82934f85ef176140ac33967ca3e355a6ee3ce88d192d3872ca7b8dee3f"} Oct 29 22:51:13.147842 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:13.147769 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2" event={"ID":"61db0858-830e-4190-8d17-6b0a41497b42","Type":"ContainerDied","Data":"0e5d932561558f4a232d9dcc9fb1ba0c1bd2876d1be4535b7f0b82f1e2b8b495"} Oct 29 22:51:13.147842 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:13.147793 4821 scope.go:122] "RemoveContainer" containerID="37e96e82934f85ef176140ac33967ca3e355a6ee3ce88d192d3872ca7b8dee3f" Oct 29 22:51:13.150464 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:13.150421 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz" event={"ID":"9809020b-a844-44fa-86af-6565434de12d","Type":"ContainerStarted","Data":"dde4eba0ad1b9a2395f591735f4880250219ee3648b9ac2d98fcaf2c02c7adc5"} Oct 29 22:51:13.150547 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:13.150473 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz" event={"ID":"9809020b-a844-44fa-86af-6565434de12d","Type":"ContainerStarted","Data":"06013545c61dcef6ebaf9ff520ccaba551305e13622c10172235908b1d0dd011"} Oct 29 22:51:13.150547 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:13.150519 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz" Oct 29 22:51:13.157900 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:13.157862 4821 scope.go:122] "RemoveContainer" containerID="71c6e470859fdc24c85a59997ddb8be30488cc9d3466bc43c00751258366dfd6" Oct 29 22:51:13.165783 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:13.165766 4821 scope.go:122] "RemoveContainer" containerID="37e96e82934f85ef176140ac33967ca3e355a6ee3ce88d192d3872ca7b8dee3f" Oct 29 22:51:13.166049 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:51:13.166029 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37e96e82934f85ef176140ac33967ca3e355a6ee3ce88d192d3872ca7b8dee3f\": container with ID starting with 37e96e82934f85ef176140ac33967ca3e355a6ee3ce88d192d3872ca7b8dee3f not found: ID does not exist" containerID="37e96e82934f85ef176140ac33967ca3e355a6ee3ce88d192d3872ca7b8dee3f" Oct 29 22:51:13.166119 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:13.166060 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37e96e82934f85ef176140ac33967ca3e355a6ee3ce88d192d3872ca7b8dee3f"} err="failed to get container status \"37e96e82934f85ef176140ac33967ca3e355a6ee3ce88d192d3872ca7b8dee3f\": rpc error: code = NotFound desc = could not find container \"37e96e82934f85ef176140ac33967ca3e355a6ee3ce88d192d3872ca7b8dee3f\": container with ID starting with 37e96e82934f85ef176140ac33967ca3e355a6ee3ce88d192d3872ca7b8dee3f not found: ID does not exist" Oct 29 22:51:13.166119 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:13.166084 4821 scope.go:122] "RemoveContainer" containerID="71c6e470859fdc24c85a59997ddb8be30488cc9d3466bc43c00751258366dfd6" Oct 29 22:51:13.166352 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:51:13.166336 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c6e470859fdc24c85a59997ddb8be30488cc9d3466bc43c00751258366dfd6\": container with ID starting with 71c6e470859fdc24c85a59997ddb8be30488cc9d3466bc43c00751258366dfd6 not found: ID does not exist" containerID="71c6e470859fdc24c85a59997ddb8be30488cc9d3466bc43c00751258366dfd6" Oct 29 22:51:13.166389 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:13.166359 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c6e470859fdc24c85a59997ddb8be30488cc9d3466bc43c00751258366dfd6"} err="failed to get container status \"71c6e470859fdc24c85a59997ddb8be30488cc9d3466bc43c00751258366dfd6\": rpc error: code = NotFound desc = could not find container \"71c6e470859fdc24c85a59997ddb8be30488cc9d3466bc43c00751258366dfd6\": container with ID starting with 71c6e470859fdc24c85a59997ddb8be30488cc9d3466bc43c00751258366dfd6 not found: ID does not exist" Oct 29 22:51:13.184770 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:13.184728 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt" podStartSLOduration=2.18471651 podStartE2EDuration="2.18471651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:13.182467858 +0000 UTC m=+527.123015768" watchObservedRunningTime="2026-10-29 22:51:13.18471651 +0000 UTC m=+527.125264419" Oct 29 22:51:13.222674 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:13.222625 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz" podStartSLOduration=1.222612001 podStartE2EDuration="1.222612001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:13.220679757 +0000 UTC m=+527.161227666" watchObservedRunningTime="2026-10-29 22:51:13.222612001 +0000 UTC m=+527.163159912" Oct 29 22:51:13.262138 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:13.262088 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podStartSLOduration=1.262073783 podStartE2EDuration="1.262073783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:13.259804935 +0000 UTC m=+527.200352846" watchObservedRunningTime="2026-10-29 22:51:13.262073783 +0000 UTC m=+527.202621692" Oct 29 22:51:13.304949 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:13.304920 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2"] Oct 29 22:51:13.308349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:13.308326 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-639067f9-gw-openshift-default-5f69bc6dd5-vdzg2"] Oct 29 22:51:13.623899 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:13.623866 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:51:14.167789 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:14.167758 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b81dd7c-d225-4645-a679-53f0c1a4d6c3" path="/var/lib/kubelet/pods/5b81dd7c-d225-4645-a679-53f0c1a4d6c3/volumes" Oct 29 22:51:14.168183 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:14.168146 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61db0858-830e-4190-8d17-6b0a41497b42" path="/var/lib/kubelet/pods/61db0858-830e-4190-8d17-6b0a41497b42/volumes" Oct 29 22:51:16.514035 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:16.513997 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f"] Oct 29 22:51:16.514534 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:16.514257 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" podUID="fc056f03-a438-4831-9803-eaa766186618" containerName="maas-api" containerID="cri-o://f48190e6defe90ac742b09dd021f80e5ad3913c06da67fc7706d49cb12acfce3" gracePeriod=30 Oct 29 22:51:16.802618 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:16.802396 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" Oct 29 22:51:16.904337 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:16.904303 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc056f03-a438-4831-9803-eaa766186618-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc056f03-a438-4831-9803-eaa766186618-openshift-service-ca\") pod \"fc056f03-a438-4831-9803-eaa766186618\" (UID: \"fc056f03-a438-4831-9803-eaa766186618\") " Oct 29 22:51:16.904519 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:16.904401 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc056f03-a438-4831-9803-eaa766186618-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc056f03-a438-4831-9803-eaa766186618-maas-api-tls\") pod \"fc056f03-a438-4831-9803-eaa766186618\" (UID: \"fc056f03-a438-4831-9803-eaa766186618\") " Oct 29 22:51:16.904519 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:16.904454 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc056f03-a438-4831-9803-eaa766186618-kube-api-access-wjq8w\" (UniqueName: \"kubernetes.io/projected/fc056f03-a438-4831-9803-eaa766186618-kube-api-access-wjq8w\") pod \"fc056f03-a438-4831-9803-eaa766186618\" (UID: \"fc056f03-a438-4831-9803-eaa766186618\") " Oct 29 22:51:16.904519 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:16.904497 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc056f03-a438-4831-9803-eaa766186618-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc056f03-a438-4831-9803-eaa766186618-metrics-certs\") pod \"fc056f03-a438-4831-9803-eaa766186618\" (UID: \"fc056f03-a438-4831-9803-eaa766186618\") " Oct 29 22:51:16.904787 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:16.904744 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc056f03-a438-4831-9803-eaa766186618-openshift-service-ca" pod "fc056f03-a438-4831-9803-eaa766186618" (UID: "fc056f03-a438-4831-9803-eaa766186618"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:16.907383 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:16.907350 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc056f03-a438-4831-9803-eaa766186618-maas-api-tls" pod "fc056f03-a438-4831-9803-eaa766186618" (UID: "fc056f03-a438-4831-9803-eaa766186618"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:51:16.907521 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:16.907379 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc056f03-a438-4831-9803-eaa766186618-kube-api-access-wjq8w" pod "fc056f03-a438-4831-9803-eaa766186618" (UID: "fc056f03-a438-4831-9803-eaa766186618"). InnerVolumeSpecName "kube-api-access-wjq8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:16.907740 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:16.907715 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc056f03-a438-4831-9803-eaa766186618-metrics-certs" pod "fc056f03-a438-4831-9803-eaa766186618" (UID: "fc056f03-a438-4831-9803-eaa766186618"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:51:17.006844 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.006805 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc056f03-a438-4831-9803-eaa766186618-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:51:17.006844 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.006839 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc056f03-a438-4831-9803-eaa766186618-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:51:17.006844 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.006850 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjq8w\" (UniqueName: \"kubernetes.io/projected/fc056f03-a438-4831-9803-eaa766186618-kube-api-access-wjq8w\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:51:17.007082 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.006859 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc056f03-a438-4831-9803-eaa766186618-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:51:17.175537 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.175507 4821 generic.go:356] "Generic (PLEG): container finished" podID="fc056f03-a438-4831-9803-eaa766186618" containerID="f48190e6defe90ac742b09dd021f80e5ad3913c06da67fc7706d49cb12acfce3" exitCode=0 Oct 29 22:51:17.175705 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.175554 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" Oct 29 22:51:17.175705 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.175597 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" event={"ID":"fc056f03-a438-4831-9803-eaa766186618","Type":"ContainerDied","Data":"f48190e6defe90ac742b09dd021f80e5ad3913c06da67fc7706d49cb12acfce3"} Oct 29 22:51:17.175705 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.175658 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f" event={"ID":"fc056f03-a438-4831-9803-eaa766186618","Type":"ContainerDied","Data":"d9e71ad4f44c12ab51c48704f72a8b891768f6268b9fa7979b0b9bc38fa37a4e"} Oct 29 22:51:17.175705 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.175689 4821 scope.go:122] "RemoveContainer" containerID="f48190e6defe90ac742b09dd021f80e5ad3913c06da67fc7706d49cb12acfce3" Oct 29 22:51:17.185732 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.185716 4821 scope.go:122] "RemoveContainer" containerID="f48190e6defe90ac742b09dd021f80e5ad3913c06da67fc7706d49cb12acfce3" Oct 29 22:51:17.186017 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:51:17.185999 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f48190e6defe90ac742b09dd021f80e5ad3913c06da67fc7706d49cb12acfce3\": container with ID starting with f48190e6defe90ac742b09dd021f80e5ad3913c06da67fc7706d49cb12acfce3 not found: ID does not exist" containerID="f48190e6defe90ac742b09dd021f80e5ad3913c06da67fc7706d49cb12acfce3" Oct 29 22:51:17.186072 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.186027 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f48190e6defe90ac742b09dd021f80e5ad3913c06da67fc7706d49cb12acfce3"} err="failed to get container status \"f48190e6defe90ac742b09dd021f80e5ad3913c06da67fc7706d49cb12acfce3\": rpc error: code = NotFound desc = could not find container \"f48190e6defe90ac742b09dd021f80e5ad3913c06da67fc7706d49cb12acfce3\": container with ID starting with f48190e6defe90ac742b09dd021f80e5ad3913c06da67fc7706d49cb12acfce3 not found: ID does not exist" Oct 29 22:51:17.237422 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.237395 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f"] Oct 29 22:51:17.243363 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.243338 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1db2136c-57968fb6dc-fz87f"] Oct 29 22:51:17.971412 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.971379 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w"] Oct 29 22:51:17.971822 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.971805 4821 cpu_manager.go:414] "containerMap: removing container" podUID="fc056f03-a438-4831-9803-eaa766186618" containerName="maas-api" Oct 29 22:51:17.971868 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.971825 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc056f03-a438-4831-9803-eaa766186618" containerName="maas-api" Oct 29 22:51:17.971868 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.971839 4821 cpu_manager.go:414] "containerMap: removing container" podUID="61db0858-830e-4190-8d17-6b0a41497b42" containerName="istio-proxy" Oct 29 22:51:17.971868 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.971845 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61db0858-830e-4190-8d17-6b0a41497b42" containerName="istio-proxy" Oct 29 22:51:17.971868 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.971856 4821 cpu_manager.go:414] "containerMap: removing container" podUID="61db0858-830e-4190-8d17-6b0a41497b42" containerName="istio-proxy" Oct 29 22:51:17.971868 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.971863 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61db0858-830e-4190-8d17-6b0a41497b42" containerName="istio-proxy" Oct 29 22:51:17.972022 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.971971 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc056f03-a438-4831-9803-eaa766186618" containerName="maas-api" Oct 29 22:51:17.972022 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.971987 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="61db0858-830e-4190-8d17-6b0a41497b42" containerName="istio-proxy" Oct 29 22:51:17.972022 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.971996 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="61db0858-830e-4190-8d17-6b0a41497b42" containerName="istio-proxy" Oct 29 22:51:17.976455 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.976432 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:51:17.992136 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:17.992115 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e131af\"/\"default-dockercfg-xwjmb\"" Oct 29 22:51:18.004238 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.004190 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e131af\"/\"simulator-w1-e131af-kserve-self-signed-certs\"" Oct 29 22:51:18.004348 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.004285 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e131af\"/\"kube-root-ca.crt\"" Oct 29 22:51:18.004348 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.004308 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e131af\"/\"openshift-service-ca.crt\"" Oct 29 22:51:18.047582 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.047552 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w"] Oct 29 22:51:18.115790 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.115759 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d3f4d01-15ca-4d75-a69b-a5adff178017-tls-certs\") pod \"simulator-w1-e131af-kserve-6c64989687-qpx6w\" (UID: \"1d3f4d01-15ca-4d75-a69b-a5adff178017\") " pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:51:18.115954 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.115814 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-home\") pod \"simulator-w1-e131af-kserve-6c64989687-qpx6w\" (UID: \"1d3f4d01-15ca-4d75-a69b-a5adff178017\") " pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:51:18.115954 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.115880 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-tmp-dir\") pod \"simulator-w1-e131af-kserve-6c64989687-qpx6w\" (UID: \"1d3f4d01-15ca-4d75-a69b-a5adff178017\") " pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:51:18.115954 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.115927 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-dshm\") pod \"simulator-w1-e131af-kserve-6c64989687-qpx6w\" (UID: \"1d3f4d01-15ca-4d75-a69b-a5adff178017\") " pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:51:18.116078 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.115997 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-model-cache\") pod \"simulator-w1-e131af-kserve-6c64989687-qpx6w\" (UID: \"1d3f4d01-15ca-4d75-a69b-a5adff178017\") " pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:51:18.116078 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.116062 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vqjv\" (UniqueName: \"kubernetes.io/projected/1d3f4d01-15ca-4d75-a69b-a5adff178017-kube-api-access-6vqjv\") pod \"simulator-w1-e131af-kserve-6c64989687-qpx6w\" (UID: \"1d3f4d01-15ca-4d75-a69b-a5adff178017\") " pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:51:18.168441 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.168408 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc056f03-a438-4831-9803-eaa766186618" path="/var/lib/kubelet/pods/fc056f03-a438-4831-9803-eaa766186618/volumes" Oct 29 22:51:18.217140 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.217112 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-model-cache\") pod \"simulator-w1-e131af-kserve-6c64989687-qpx6w\" (UID: \"1d3f4d01-15ca-4d75-a69b-a5adff178017\") " pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:51:18.217333 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.217162 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vqjv\" (UniqueName: \"kubernetes.io/projected/1d3f4d01-15ca-4d75-a69b-a5adff178017-kube-api-access-6vqjv\") pod \"simulator-w1-e131af-kserve-6c64989687-qpx6w\" (UID: \"1d3f4d01-15ca-4d75-a69b-a5adff178017\") " pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:51:18.217333 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.217193 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d3f4d01-15ca-4d75-a69b-a5adff178017-tls-certs\") pod \"simulator-w1-e131af-kserve-6c64989687-qpx6w\" (UID: \"1d3f4d01-15ca-4d75-a69b-a5adff178017\") " pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:51:18.217333 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.217244 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-home\") pod \"simulator-w1-e131af-kserve-6c64989687-qpx6w\" (UID: \"1d3f4d01-15ca-4d75-a69b-a5adff178017\") " pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:51:18.217333 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.217262 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-tmp-dir\") pod \"simulator-w1-e131af-kserve-6c64989687-qpx6w\" (UID: \"1d3f4d01-15ca-4d75-a69b-a5adff178017\") " pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:51:18.217333 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.217298 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-dshm\") pod \"simulator-w1-e131af-kserve-6c64989687-qpx6w\" (UID: \"1d3f4d01-15ca-4d75-a69b-a5adff178017\") " pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:51:18.217612 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.217591 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-model-cache\") pod \"simulator-w1-e131af-kserve-6c64989687-qpx6w\" (UID: \"1d3f4d01-15ca-4d75-a69b-a5adff178017\") " pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:51:18.217668 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.217627 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-tmp-dir\") pod \"simulator-w1-e131af-kserve-6c64989687-qpx6w\" (UID: \"1d3f4d01-15ca-4d75-a69b-a5adff178017\") " pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:51:18.217726 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.217676 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-home\") pod \"simulator-w1-e131af-kserve-6c64989687-qpx6w\" (UID: \"1d3f4d01-15ca-4d75-a69b-a5adff178017\") " pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:51:18.219635 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.219606 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-dshm\") pod \"simulator-w1-e131af-kserve-6c64989687-qpx6w\" (UID: \"1d3f4d01-15ca-4d75-a69b-a5adff178017\") " pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:51:18.219984 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.219964 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d3f4d01-15ca-4d75-a69b-a5adff178017-tls-certs\") pod \"simulator-w1-e131af-kserve-6c64989687-qpx6w\" (UID: \"1d3f4d01-15ca-4d75-a69b-a5adff178017\") " pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:51:18.239806 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.239749 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vqjv\" (UniqueName: \"kubernetes.io/projected/1d3f4d01-15ca-4d75-a69b-a5adff178017-kube-api-access-6vqjv\") pod \"simulator-w1-e131af-kserve-6c64989687-qpx6w\" (UID: \"1d3f4d01-15ca-4d75-a69b-a5adff178017\") " pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:51:18.288714 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.288677 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:51:18.602962 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.602939 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w"] Oct 29 22:51:18.605407 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:51:18.605365 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d3f4d01_15ca_4d75_a69b_a5adff178017.slice/crio-b1e726d9dcbb2e5e54a70666cbf073d5fc16499a33aeac75862bc5b8e6d22a98 WatchSource:0}: Error finding container b1e726d9dcbb2e5e54a70666cbf073d5fc16499a33aeac75862bc5b8e6d22a98: Status 404 returned error can't find the container with id b1e726d9dcbb2e5e54a70666cbf073d5fc16499a33aeac75862bc5b8e6d22a98 Oct 29 22:51:18.716189 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.716153 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp"] Oct 29 22:51:18.720505 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.720476 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.738655 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.738620 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-575d59-openshift-default-dockercfg-gzz6n\"" Oct 29 22:51:18.755125 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.755085 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp"] Oct 29 22:51:18.823164 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.823128 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.823164 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.823160 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7874a224-3fab-4fee-90db-89b753ed938f-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.823416 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.823182 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.823416 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.823290 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.823416 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.823327 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.823416 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.823345 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.823609 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.823456 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7874a224-3fab-4fee-90db-89b753ed938f-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.823609 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.823536 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9d2n\" (UniqueName: \"kubernetes.io/projected/7874a224-3fab-4fee-90db-89b753ed938f-kube-api-access-p9d2n\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.823609 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.823583 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7874a224-3fab-4fee-90db-89b753ed938f-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.926769 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.926730 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.926769 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.926780 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7874a224-3fab-4fee-90db-89b753ed938f-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.927060 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.926810 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.927060 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.926865 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.927060 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.926923 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.927060 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.926949 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.927060 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.927009 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7874a224-3fab-4fee-90db-89b753ed938f-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.927060 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.927035 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9d2n\" (UniqueName: \"kubernetes.io/projected/7874a224-3fab-4fee-90db-89b753ed938f-kube-api-access-p9d2n\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.927351 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.927063 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7874a224-3fab-4fee-90db-89b753ed938f-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.928143 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.928114 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7874a224-3fab-4fee-90db-89b753ed938f-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.928447 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.928426 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.929031 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.928998 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.929280 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.929257 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.930455 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.930361 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.932065 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.932034 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.932467 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.932447 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7874a224-3fab-4fee-90db-89b753ed938f-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.951479 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.951452 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7874a224-3fab-4fee-90db-89b753ed938f-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:18.955312 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:18.955286 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9d2n\" (UniqueName: \"kubernetes.io/projected/7874a224-3fab-4fee-90db-89b753ed938f-kube-api-access-p9d2n\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:19.037105 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:19.037070 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:19.191155 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:19.191051 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" event={"ID":"1d3f4d01-15ca-4d75-a69b-a5adff178017","Type":"ContainerStarted","Data":"abbdb33664418d4d2f2d2b3ffc4859dda93e43aa0d1dbcddaf78c0818dcd9fef"} Oct 29 22:51:19.191155 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:19.191105 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" event={"ID":"1d3f4d01-15ca-4d75-a69b-a5adff178017","Type":"ContainerStarted","Data":"b1e726d9dcbb2e5e54a70666cbf073d5fc16499a33aeac75862bc5b8e6d22a98"} Oct 29 22:51:19.233663 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:19.233615 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podStartSLOduration=2.233601137 podStartE2EDuration="2.233601137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:19.231954909 +0000 UTC m=+533.172502819" watchObservedRunningTime="2026-10-29 22:51:19.233601137 +0000 UTC m=+533.174149085" Oct 29 22:51:19.288846 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:19.288819 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:51:19.363982 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:19.363945 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp"] Oct 29 22:51:19.368807 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:51:19.368779 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7874a224_3fab_4fee_90db_89b753ed938f.slice/crio-f109d3915487834276eb9fc6b3263a1108513e51768f06b0591929279ae7620e WatchSource:0}: Error finding container f109d3915487834276eb9fc6b3263a1108513e51768f06b0591929279ae7620e: Status 404 returned error can't find the container with id f109d3915487834276eb9fc6b3263a1108513e51768f06b0591929279ae7620e Oct 29 22:51:19.372143 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:19.372112 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:51:19.372254 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:19.372175 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:51:19.372254 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:19.372239 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:51:20.197974 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:20.197934 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" event={"ID":"7874a224-3fab-4fee-90db-89b753ed938f","Type":"ContainerStarted","Data":"b5fdd38111e89a63a92f4d03e43c5fd334d1d0f893220d07aa97b24ee00462bf"} Oct 29 22:51:20.198417 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:20.197995 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" event={"ID":"7874a224-3fab-4fee-90db-89b753ed938f","Type":"ContainerStarted","Data":"f109d3915487834276eb9fc6b3263a1108513e51768f06b0591929279ae7620e"} Oct 29 22:51:20.254592 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:20.254538 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" podStartSLOduration=2.2545245769999998 podStartE2EDuration="2.254524577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:20.252814363 +0000 UTC m=+534.193362264" watchObservedRunningTime="2026-10-29 22:51:20.254524577 +0000 UTC m=+534.195072486" Oct 29 22:51:21.038100 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:21.038068 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:21.043541 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:21.043509 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:21.202985 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:21.202949 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:21.203820 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:21.203796 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:22.624743 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:22.624702 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:51:22.638190 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:22.638162 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:51:23.228235 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:23.228178 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:51:23.884483 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:51:23.884442 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:51:24.159282 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:24.159177 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt" Oct 29 22:51:24.159436 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:24.159382 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz" Oct 29 22:51:28.289661 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:28.289627 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:51:28.302648 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:28.302621 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:51:29.297227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:29.286118 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:51:33.622127 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:33.622093 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn"] Oct 29 22:51:33.628017 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:33.627979 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:51:33.653825 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:33.653801 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-08e15d\"/\"premium-w0-08e15d-kserve-self-signed-certs\"" Oct 29 22:51:33.656048 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:33.656026 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn"] Oct 29 22:51:33.672914 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:33.672884 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-tmp-dir\") pod \"premium-w0-08e15d-kserve-7cbc4d9bc-frwmn\" (UID: \"28f5dfd6-1e9c-447d-ac89-82393448335d\") " pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:51:33.673031 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:33.672938 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-dshm\") pod \"premium-w0-08e15d-kserve-7cbc4d9bc-frwmn\" (UID: \"28f5dfd6-1e9c-447d-ac89-82393448335d\") " pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:51:33.673105 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:33.673020 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-model-cache\") pod \"premium-w0-08e15d-kserve-7cbc4d9bc-frwmn\" (UID: \"28f5dfd6-1e9c-447d-ac89-82393448335d\") " pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:51:33.673105 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:33.673067 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-home\") pod \"premium-w0-08e15d-kserve-7cbc4d9bc-frwmn\" (UID: \"28f5dfd6-1e9c-447d-ac89-82393448335d\") " pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:51:33.673274 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:33.673249 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kprln\" (UniqueName: \"kubernetes.io/projected/28f5dfd6-1e9c-447d-ac89-82393448335d-kube-api-access-kprln\") pod \"premium-w0-08e15d-kserve-7cbc4d9bc-frwmn\" (UID: \"28f5dfd6-1e9c-447d-ac89-82393448335d\") " pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:51:33.673333 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:33.673310 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28f5dfd6-1e9c-447d-ac89-82393448335d-tls-certs\") pod \"premium-w0-08e15d-kserve-7cbc4d9bc-frwmn\" (UID: \"28f5dfd6-1e9c-447d-ac89-82393448335d\") " pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:51:33.774351 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:33.774304 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kprln\" (UniqueName: \"kubernetes.io/projected/28f5dfd6-1e9c-447d-ac89-82393448335d-kube-api-access-kprln\") pod \"premium-w0-08e15d-kserve-7cbc4d9bc-frwmn\" (UID: \"28f5dfd6-1e9c-447d-ac89-82393448335d\") " pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:51:33.774545 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:33.774391 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28f5dfd6-1e9c-447d-ac89-82393448335d-tls-certs\") pod \"premium-w0-08e15d-kserve-7cbc4d9bc-frwmn\" (UID: \"28f5dfd6-1e9c-447d-ac89-82393448335d\") " pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:51:33.774545 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:33.774505 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-tmp-dir\") pod \"premium-w0-08e15d-kserve-7cbc4d9bc-frwmn\" (UID: \"28f5dfd6-1e9c-447d-ac89-82393448335d\") " pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:51:33.774545 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:33.774536 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-dshm\") pod \"premium-w0-08e15d-kserve-7cbc4d9bc-frwmn\" (UID: \"28f5dfd6-1e9c-447d-ac89-82393448335d\") " pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:51:33.774705 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:33.774561 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-model-cache\") pod \"premium-w0-08e15d-kserve-7cbc4d9bc-frwmn\" (UID: \"28f5dfd6-1e9c-447d-ac89-82393448335d\") " pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:51:33.774705 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:33.774597 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-home\") pod \"premium-w0-08e15d-kserve-7cbc4d9bc-frwmn\" (UID: \"28f5dfd6-1e9c-447d-ac89-82393448335d\") " pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:51:33.775013 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:33.774984 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-model-cache\") pod \"premium-w0-08e15d-kserve-7cbc4d9bc-frwmn\" (UID: \"28f5dfd6-1e9c-447d-ac89-82393448335d\") " pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:51:33.775087 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:33.774996 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-home\") pod \"premium-w0-08e15d-kserve-7cbc4d9bc-frwmn\" (UID: \"28f5dfd6-1e9c-447d-ac89-82393448335d\") " pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:51:33.775342 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:33.775318 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-tmp-dir\") pod \"premium-w0-08e15d-kserve-7cbc4d9bc-frwmn\" (UID: \"28f5dfd6-1e9c-447d-ac89-82393448335d\") " pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:51:33.776947 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:33.776925 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-dshm\") pod \"premium-w0-08e15d-kserve-7cbc4d9bc-frwmn\" (UID: \"28f5dfd6-1e9c-447d-ac89-82393448335d\") " pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:51:33.777442 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:33.777417 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28f5dfd6-1e9c-447d-ac89-82393448335d-tls-certs\") pod \"premium-w0-08e15d-kserve-7cbc4d9bc-frwmn\" (UID: \"28f5dfd6-1e9c-447d-ac89-82393448335d\") " pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:51:33.797842 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:33.797811 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kprln\" (UniqueName: \"kubernetes.io/projected/28f5dfd6-1e9c-447d-ac89-82393448335d-kube-api-access-kprln\") pod \"premium-w0-08e15d-kserve-7cbc4d9bc-frwmn\" (UID: \"28f5dfd6-1e9c-447d-ac89-82393448335d\") " pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:51:33.940455 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:33.940374 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:51:34.243518 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:34.243480 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn"] Oct 29 22:51:34.245503 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:51:34.245474 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28f5dfd6_1e9c_447d_ac89_82393448335d.slice/crio-e329d06a76eaab17cae62d2b1af2e0c76ceb093dd3fc3b99c41e0ce4ea4b130b WatchSource:0}: Error finding container e329d06a76eaab17cae62d2b1af2e0c76ceb093dd3fc3b99c41e0ce4ea4b130b: Status 404 returned error can't find the container with id e329d06a76eaab17cae62d2b1af2e0c76ceb093dd3fc3b99c41e0ce4ea4b130b Oct 29 22:51:34.287111 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:34.287088 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" event={"ID":"28f5dfd6-1e9c-447d-ac89-82393448335d","Type":"ContainerStarted","Data":"e329d06a76eaab17cae62d2b1af2e0c76ceb093dd3fc3b99c41e0ce4ea4b130b"} Oct 29 22:51:35.294084 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.294043 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" event={"ID":"28f5dfd6-1e9c-447d-ac89-82393448335d","Type":"ContainerStarted","Data":"1f87c1f07c414a80463890155eb3a5dc315ad4848538f803f3e7965a597766f7"} Oct 29 22:51:35.338722 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.338666 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podStartSLOduration=2.338651581 podStartE2EDuration="2.338651581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:35.337393307 +0000 UTC m=+549.277941216" watchObservedRunningTime="2026-10-29 22:51:35.338651581 +0000 UTC m=+549.279199491" Oct 29 22:51:35.351578 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.351546 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84"] Oct 29 22:51:35.355388 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.355368 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.372354 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.372334 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-31226c-openshift-default-dockercfg-cmhxj\"" Oct 29 22:51:35.382800 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.382770 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84"] Oct 29 22:51:35.390338 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.390313 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.390455 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.390344 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.390455 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.390420 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m2h7\" (UniqueName: \"kubernetes.io/projected/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-kube-api-access-6m2h7\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.390537 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.390477 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.390537 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.390530 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.390611 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.390591 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.390734 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.390714 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.390775 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.390755 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.390809 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.390779 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.491875 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.491826 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.491875 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.491879 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.492127 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.491910 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m2h7\" (UniqueName: \"kubernetes.io/projected/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-kube-api-access-6m2h7\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.492127 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.491933 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.492127 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.491964 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.492127 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.491998 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.492127 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.492092 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.492127 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.492123 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.492472 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.492153 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.492472 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.492309 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.492472 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.492343 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.492639 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.492519 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.492639 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.492606 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.493159 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.492935 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.494414 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.494388 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.494714 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.494695 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.514180 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.514155 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.514345 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.514328 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m2h7\" (UniqueName: \"kubernetes.io/projected/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-kube-api-access-6m2h7\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.668707 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.668667 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:35.941531 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.941460 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:51:35.996421 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.990714 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84"] Oct 29 22:51:35.998687 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.998654 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:51:35.998781 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.998718 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:51:35.998781 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:35.998746 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:51:36.301453 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:36.301367 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" event={"ID":"e6f4fd46-5aab-49a8-9446-f71ed103d2eb","Type":"ContainerStarted","Data":"f49f74bb323fdca15e5b27b1a0298fbbbea9b4e670b9c83fd9c6062441e33602"} Oct 29 22:51:36.301453 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:36.301428 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" event={"ID":"e6f4fd46-5aab-49a8-9446-f71ed103d2eb","Type":"ContainerStarted","Data":"bea33b5f18c7ed547384ac035f02e14fe69bea76c1fe5f35236dc0631db46509"} Oct 29 22:51:36.348637 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:36.348575 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" podStartSLOduration=1.348554816 podStartE2EDuration="1.348554816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:36.345768655 +0000 UTC m=+550.286316578" watchObservedRunningTime="2026-10-29 22:51:36.348554816 +0000 UTC m=+550.289102728" Oct 29 22:51:36.669655 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:36.669617 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:37.671091 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:37.671049 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 29 22:51:37.671626 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:37.671107 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" podUID="e6f4fd46-5aab-49a8-9446-f71ed103d2eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 29 22:51:38.669908 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:38.669872 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 29 22:51:38.670094 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:38.669923 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" podUID="e6f4fd46-5aab-49a8-9446-f71ed103d2eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 29 22:51:39.670542 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:39.670501 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 29 22:51:39.671035 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:39.670584 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" podUID="e6f4fd46-5aab-49a8-9446-f71ed103d2eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 29 22:51:40.669812 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:40.669772 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 29 22:51:40.669975 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:40.669848 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" podUID="e6f4fd46-5aab-49a8-9446-f71ed103d2eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 29 22:51:41.670617 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:41.670576 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 29 22:51:41.671726 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:41.671308 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" podUID="e6f4fd46-5aab-49a8-9446-f71ed103d2eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 29 22:51:43.001230 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:42.999253 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:43.001230 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:43.000391 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:43.001230 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:43.001087 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 22:51:43.943033 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:43.942996 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:51:43.957439 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:43.957412 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:51:44.339565 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.339465 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp"] Oct 29 22:51:44.344692 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.344665 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.362180 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.362147 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e131af-openshift-default-dockercfg-796wj\"" Oct 29 22:51:44.376563 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.376535 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp"] Oct 29 22:51:44.453097 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.453065 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:51:44.453686 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.453661 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.453810 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.453708 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.453810 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.453733 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpcnw\" (UniqueName: \"kubernetes.io/projected/23d4c06e-ed49-45da-85d5-55e60a3d48c1-kube-api-access-vpcnw\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.453885 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.453830 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.453926 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.453883 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.453983 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.453967 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.454020 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.453997 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.454065 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.454029 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.454153 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.454137 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.555854 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.555812 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.556225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.556178 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.556446 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.556427 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.556606 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.556582 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.556717 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.556584 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.556785 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.556755 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.556848 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.556834 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.556848 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.556838 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.556952 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.556895 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.556952 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.556922 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpcnw\" (UniqueName: \"kubernetes.io/projected/23d4c06e-ed49-45da-85d5-55e60a3d48c1-kube-api-access-vpcnw\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.557063 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.556961 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.557122 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.557096 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.557679 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.557657 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.557844 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.557821 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.558700 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.558659 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.559194 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.559168 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.580186 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.580162 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.580321 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.580262 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpcnw\" (UniqueName: \"kubernetes.io/projected/23d4c06e-ed49-45da-85d5-55e60a3d48c1-kube-api-access-vpcnw\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.659349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.659315 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:44.997010 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:44.996981 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp"] Oct 29 22:51:45.000702 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:45.000667 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:51:45.000827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:45.000748 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:51:45.000827 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:45.000795 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:51:45.449800 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:45.449745 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" event={"ID":"23d4c06e-ed49-45da-85d5-55e60a3d48c1","Type":"ContainerStarted","Data":"2624cf98aad5c6b0206f8aed0e6085196b60a46c21174d632fdcdf6d8bb53e0b"} Oct 29 22:51:45.450282 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:45.449828 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" event={"ID":"23d4c06e-ed49-45da-85d5-55e60a3d48c1","Type":"ContainerStarted","Data":"d34c1e3326437327fea9676687273e1ee48cf75ff1dbd5421fde1f63fdd4fcf9"} Oct 29 22:51:45.495531 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:45.495467 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" podStartSLOduration=1.495452415 podStartE2EDuration="1.495452415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:45.493759877 +0000 UTC m=+559.434307788" watchObservedRunningTime="2026-10-29 22:51:45.495452415 +0000 UTC m=+559.436000324" Oct 29 22:51:45.659559 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:45.659520 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:46.661256 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:46.661191 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 29 22:51:46.661674 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:46.661282 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" podUID="23d4c06e-ed49-45da-85d5-55e60a3d48c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 29 22:51:47.659665 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:47.659619 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 29 22:51:47.659872 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:47.659669 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" podUID="23d4c06e-ed49-45da-85d5-55e60a3d48c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 29 22:51:48.659695 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:48.659656 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 29 22:51:48.660085 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:48.659714 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" podUID="23d4c06e-ed49-45da-85d5-55e60a3d48c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 29 22:51:49.478953 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:49.478919 4821 generic.go:356] "Generic (PLEG): container finished" podID="7874a224-3fab-4fee-90db-89b753ed938f" containerID="b5fdd38111e89a63a92f4d03e43c5fd334d1d0f893220d07aa97b24ee00462bf" exitCode=0 Oct 29 22:51:49.479157 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:49.478994 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" event={"ID":"7874a224-3fab-4fee-90db-89b753ed938f","Type":"ContainerDied","Data":"b5fdd38111e89a63a92f4d03e43c5fd334d1d0f893220d07aa97b24ee00462bf"} Oct 29 22:51:49.479404 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:49.479390 4821 scope.go:122] "RemoveContainer" containerID="b5fdd38111e89a63a92f4d03e43c5fd334d1d0f893220d07aa97b24ee00462bf" Oct 29 22:51:49.480669 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:49.480646 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:51:49.480737 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:49.480711 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:51:49.480773 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:49.480740 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:51:49.659935 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:49.659898 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 29 22:51:49.660368 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:49.659974 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" podUID="23d4c06e-ed49-45da-85d5-55e60a3d48c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 29 22:51:50.487427 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:50.487392 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" event={"ID":"7874a224-3fab-4fee-90db-89b753ed938f","Type":"ContainerStarted","Data":"af11dc8eb42c2bc59c9001037e0df69864aaccb9b5ca5eb4b39eb4b63f125893"} Oct 29 22:51:50.664665 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:50.664619 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 29 22:51:50.665184 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:50.664681 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" podUID="23d4c06e-ed49-45da-85d5-55e60a3d48c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 29 22:51:51.039225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:51.038511 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:51.039225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:51.038682 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 29 22:51:51.039225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:51.038727 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" podUID="7874a224-3fab-4fee-90db-89b753ed938f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 29 22:51:51.203302 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:51.203270 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:51.659688 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:51.659647 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 29 22:51:51.659899 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:51.659715 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" podUID="23d4c06e-ed49-45da-85d5-55e60a3d48c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 29 22:51:52.037764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:52.037653 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 29 22:51:52.038408 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:52.038374 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" podUID="7874a224-3fab-4fee-90db-89b753ed938f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 29 22:51:52.664852 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:52.664824 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:52.665252 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:52.665233 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:52.666051 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:52.666016 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 22:51:53.037740 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.037647 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 29 22:51:53.038036 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.037990 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" podUID="7874a224-3fab-4fee-90db-89b753ed938f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 29 22:51:53.280112 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.280049 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n"] Oct 29 22:51:53.290697 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.290610 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:51:53.347225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.316499 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n"] Oct 29 22:51:53.387241 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.379597 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-17394a\"/\"default-dockercfg-mtks9\"" Oct 29 22:51:53.387241 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.379725 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-17394a\"/\"kube-root-ca.crt\"" Oct 29 22:51:53.387241 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.381350 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-17394a\"/\"openshift-service-ca.crt\"" Oct 29 22:51:53.387241 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.381508 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-17394a\"/\"simulator-w3-17394a-kserve-self-signed-certs\"" Oct 29 22:51:53.395857 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.395821 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9g6v\" (UniqueName: \"kubernetes.io/projected/e23010a6-7ea5-40e3-b67d-f88d04c30e81-kube-api-access-n9g6v\") pod \"simulator-w3-17394a-kserve-899fdb57f-s526n\" (UID: \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\") " pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:51:53.396085 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.396064 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-model-cache\") pod \"simulator-w3-17394a-kserve-899fdb57f-s526n\" (UID: \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\") " pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:51:53.396258 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.396239 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-dshm\") pod \"simulator-w3-17394a-kserve-899fdb57f-s526n\" (UID: \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\") " pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:51:53.396743 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.396401 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-home\") pod \"simulator-w3-17394a-kserve-899fdb57f-s526n\" (UID: \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\") " pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:51:53.396743 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.396475 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23010a6-7ea5-40e3-b67d-f88d04c30e81-tls-certs\") pod \"simulator-w3-17394a-kserve-899fdb57f-s526n\" (UID: \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\") " pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:51:53.396743 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.396521 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-tmp-dir\") pod \"simulator-w3-17394a-kserve-899fdb57f-s526n\" (UID: \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\") " pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:51:53.515291 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.502747 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-tmp-dir\") pod \"simulator-w3-17394a-kserve-899fdb57f-s526n\" (UID: \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\") " pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:51:53.515291 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.502880 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9g6v\" (UniqueName: \"kubernetes.io/projected/e23010a6-7ea5-40e3-b67d-f88d04c30e81-kube-api-access-n9g6v\") pod \"simulator-w3-17394a-kserve-899fdb57f-s526n\" (UID: \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\") " pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:51:53.515291 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.502913 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-model-cache\") pod \"simulator-w3-17394a-kserve-899fdb57f-s526n\" (UID: \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\") " pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:51:53.515291 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.502959 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-dshm\") pod \"simulator-w3-17394a-kserve-899fdb57f-s526n\" (UID: \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\") " pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:51:53.515291 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.502983 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-home\") pod \"simulator-w3-17394a-kserve-899fdb57f-s526n\" (UID: \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\") " pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:51:53.515291 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.503039 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23010a6-7ea5-40e3-b67d-f88d04c30e81-tls-certs\") pod \"simulator-w3-17394a-kserve-899fdb57f-s526n\" (UID: \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\") " pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:51:53.515291 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.503961 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-tmp-dir\") pod \"simulator-w3-17394a-kserve-899fdb57f-s526n\" (UID: \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\") " pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:51:53.528239 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.528187 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-dshm\") pod \"simulator-w3-17394a-kserve-899fdb57f-s526n\" (UID: \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\") " pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:51:53.529247 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.529224 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-home\") pod \"simulator-w3-17394a-kserve-899fdb57f-s526n\" (UID: \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\") " pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:51:53.546867 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.546801 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-model-cache\") pod \"simulator-w3-17394a-kserve-899fdb57f-s526n\" (UID: \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\") " pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:51:53.547719 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.547655 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23010a6-7ea5-40e3-b67d-f88d04c30e81-tls-certs\") pod \"simulator-w3-17394a-kserve-899fdb57f-s526n\" (UID: \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\") " pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:51:53.578924 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.578891 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9g6v\" (UniqueName: \"kubernetes.io/projected/e23010a6-7ea5-40e3-b67d-f88d04c30e81-kube-api-access-n9g6v\") pod \"simulator-w3-17394a-kserve-899fdb57f-s526n\" (UID: \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\") " pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:51:53.691709 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:53.691407 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:51:53.971540 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:51:53.971501 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:51:54.041111 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:54.040651 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 29 22:51:54.041111 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:54.040711 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" podUID="7874a224-3fab-4fee-90db-89b753ed938f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 29 22:51:54.173541 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:54.173516 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n"] Oct 29 22:51:54.175047 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:51:54.175022 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode23010a6_7ea5_40e3_b67d_f88d04c30e81.slice/crio-3d77d759ac415f0c433e4ec29df199a4e247c72f6ba91959300d799271d4f3df WatchSource:0}: Error finding container 3d77d759ac415f0c433e4ec29df199a4e247c72f6ba91959300d799271d4f3df: Status 404 returned error can't find the container with id 3d77d759ac415f0c433e4ec29df199a4e247c72f6ba91959300d799271d4f3df Oct 29 22:51:54.561910 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:54.561808 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" event={"ID":"e23010a6-7ea5-40e3-b67d-f88d04c30e81","Type":"ContainerStarted","Data":"f7593facdec14c6344cbe725bc09100f14ab15f7b00d27f159d3dfc51147e91d"} Oct 29 22:51:54.561910 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:54.561851 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" event={"ID":"e23010a6-7ea5-40e3-b67d-f88d04c30e81","Type":"ContainerStarted","Data":"3d77d759ac415f0c433e4ec29df199a4e247c72f6ba91959300d799271d4f3df"} Oct 29 22:51:54.607285 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:54.607193 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podStartSLOduration=1.6071747140000001 podStartE2EDuration="1.607174714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:54.604597801 +0000 UTC m=+568.545145724" watchObservedRunningTime="2026-10-29 22:51:54.607174714 +0000 UTC m=+568.547722625" Oct 29 22:51:54.693012 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:54.692980 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:51:55.039230 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:55.037943 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 29 22:51:55.039230 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:55.038002 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" podUID="7874a224-3fab-4fee-90db-89b753ed938f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 29 22:51:56.041555 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:56.041525 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:51:56.042583 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:51:56.042562 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 22:52:03.692069 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:03.692031 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:52:03.704867 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:03.704844 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:52:04.631186 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:04.631155 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:52:13.108981 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.108947 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4"] Oct 29 22:52:13.112977 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.112943 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:52:13.138831 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.138797 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-b2921b\"/\"kube-root-ca.crt\"" Oct 29 22:52:13.138985 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.138835 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b2921b\"/\"simulator-w5-b2921b-kserve-self-signed-certs\"" Oct 29 22:52:13.138985 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.138912 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b2921b\"/\"default-dockercfg-b7z2q\"" Oct 29 22:52:13.139106 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.139080 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-b2921b\"/\"openshift-service-ca.crt\"" Oct 29 22:52:13.141353 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.141330 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4"] Oct 29 22:52:13.143362 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.143336 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-dshm\") pod \"simulator-w5-b2921b-kserve-5555547b64-xs9h4\" (UID: \"28221151-2a2b-46b8-9fee-b658ee323a32\") " pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:52:13.143479 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.143455 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dtwt\" (UniqueName: \"kubernetes.io/projected/28221151-2a2b-46b8-9fee-b658ee323a32-kube-api-access-7dtwt\") pod \"simulator-w5-b2921b-kserve-5555547b64-xs9h4\" (UID: \"28221151-2a2b-46b8-9fee-b658ee323a32\") " pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:52:13.143567 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.143488 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-model-cache\") pod \"simulator-w5-b2921b-kserve-5555547b64-xs9h4\" (UID: \"28221151-2a2b-46b8-9fee-b658ee323a32\") " pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:52:13.143567 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.143520 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-tmp-dir\") pod \"simulator-w5-b2921b-kserve-5555547b64-xs9h4\" (UID: \"28221151-2a2b-46b8-9fee-b658ee323a32\") " pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:52:13.143567 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.143550 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28221151-2a2b-46b8-9fee-b658ee323a32-tls-certs\") pod \"simulator-w5-b2921b-kserve-5555547b64-xs9h4\" (UID: \"28221151-2a2b-46b8-9fee-b658ee323a32\") " pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:52:13.143753 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.143668 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-home\") pod \"simulator-w5-b2921b-kserve-5555547b64-xs9h4\" (UID: \"28221151-2a2b-46b8-9fee-b658ee323a32\") " pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:52:13.245086 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.245041 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dtwt\" (UniqueName: \"kubernetes.io/projected/28221151-2a2b-46b8-9fee-b658ee323a32-kube-api-access-7dtwt\") pod \"simulator-w5-b2921b-kserve-5555547b64-xs9h4\" (UID: \"28221151-2a2b-46b8-9fee-b658ee323a32\") " pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:52:13.245086 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.245085 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-model-cache\") pod \"simulator-w5-b2921b-kserve-5555547b64-xs9h4\" (UID: \"28221151-2a2b-46b8-9fee-b658ee323a32\") " pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:52:13.245383 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.245108 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-tmp-dir\") pod \"simulator-w5-b2921b-kserve-5555547b64-xs9h4\" (UID: \"28221151-2a2b-46b8-9fee-b658ee323a32\") " pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:52:13.245383 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.245143 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28221151-2a2b-46b8-9fee-b658ee323a32-tls-certs\") pod \"simulator-w5-b2921b-kserve-5555547b64-xs9h4\" (UID: \"28221151-2a2b-46b8-9fee-b658ee323a32\") " pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:52:13.245383 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.245169 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-home\") pod \"simulator-w5-b2921b-kserve-5555547b64-xs9h4\" (UID: \"28221151-2a2b-46b8-9fee-b658ee323a32\") " pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:52:13.245383 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.245254 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-dshm\") pod \"simulator-w5-b2921b-kserve-5555547b64-xs9h4\" (UID: \"28221151-2a2b-46b8-9fee-b658ee323a32\") " pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:52:13.245678 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.245627 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-tmp-dir\") pod \"simulator-w5-b2921b-kserve-5555547b64-xs9h4\" (UID: \"28221151-2a2b-46b8-9fee-b658ee323a32\") " pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:52:13.245811 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.245686 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-home\") pod \"simulator-w5-b2921b-kserve-5555547b64-xs9h4\" (UID: \"28221151-2a2b-46b8-9fee-b658ee323a32\") " pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:52:13.245992 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.245967 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-model-cache\") pod \"simulator-w5-b2921b-kserve-5555547b64-xs9h4\" (UID: \"28221151-2a2b-46b8-9fee-b658ee323a32\") " pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:52:13.248164 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.248138 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-dshm\") pod \"simulator-w5-b2921b-kserve-5555547b64-xs9h4\" (UID: \"28221151-2a2b-46b8-9fee-b658ee323a32\") " pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:52:13.248486 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.248462 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28221151-2a2b-46b8-9fee-b658ee323a32-tls-certs\") pod \"simulator-w5-b2921b-kserve-5555547b64-xs9h4\" (UID: \"28221151-2a2b-46b8-9fee-b658ee323a32\") " pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:52:13.269138 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.269115 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dtwt\" (UniqueName: \"kubernetes.io/projected/28221151-2a2b-46b8-9fee-b658ee323a32-kube-api-access-7dtwt\") pod \"simulator-w5-b2921b-kserve-5555547b64-xs9h4\" (UID: \"28221151-2a2b-46b8-9fee-b658ee323a32\") " pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:52:13.428322 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.428226 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:52:13.737690 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:13.737665 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4"] Oct 29 22:52:13.739741 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:52:13.739716 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28221151_2a2b_46b8_9fee_b658ee323a32.slice/crio-02ba8db9510b89fd73a0bb83714a7913a8eec19072cbd0c9bcabb5d1d9deb1ef WatchSource:0}: Error finding container 02ba8db9510b89fd73a0bb83714a7913a8eec19072cbd0c9bcabb5d1d9deb1ef: Status 404 returned error can't find the container with id 02ba8db9510b89fd73a0bb83714a7913a8eec19072cbd0c9bcabb5d1d9deb1ef Oct 29 22:52:14.704166 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:14.701955 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" event={"ID":"28221151-2a2b-46b8-9fee-b658ee323a32","Type":"ContainerStarted","Data":"8985552fa7aa7bbb87b988b353997db125461d9a52f87deacaff33c4bfed7d6c"} Oct 29 22:52:14.704166 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:14.702013 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" event={"ID":"28221151-2a2b-46b8-9fee-b658ee323a32","Type":"ContainerStarted","Data":"02ba8db9510b89fd73a0bb83714a7913a8eec19072cbd0c9bcabb5d1d9deb1ef"} Oct 29 22:52:14.760137 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:14.760075 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podStartSLOduration=1.760053399 podStartE2EDuration="1.760053399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:14.756160128 +0000 UTC m=+588.696708041" watchObservedRunningTime="2026-10-29 22:52:14.760053399 +0000 UTC m=+588.700601310" Oct 29 22:52:15.428708 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:15.428661 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:52:16.210163 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:16.210129 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea5e601-k8zn9"] Oct 29 22:52:16.213671 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:16.213652 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea5e601-k8zn9" Oct 29 22:52:16.228701 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:16.228680 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:52:16.240534 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:16.240506 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 22:52:16.243057 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:16.243036 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea5e601-k8zn9"] Oct 29 22:52:16.280995 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:16.280968 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867af907-22c0-4265-bc64-576cc553de73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ea5e601-k8zn9\" (UID: \"867af907-22c0-4265-bc64-576cc553de73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea5e601-k8zn9" Oct 29 22:52:16.281153 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:16.281022 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867af907-22c0-4265-bc64-576cc553de73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ea5e601-k8zn9\" (UID: \"867af907-22c0-4265-bc64-576cc553de73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea5e601-k8zn9" Oct 29 22:52:16.281252 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:16.281192 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szk6n\" (UniqueName: \"kubernetes.io/projected/867af907-22c0-4265-bc64-576cc553de73-kube-api-access-szk6n\") pod \"maas-api-revoke-sub-2ea5e601-k8zn9\" (UID: \"867af907-22c0-4265-bc64-576cc553de73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea5e601-k8zn9" Oct 29 22:52:16.382512 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:16.382476 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867af907-22c0-4265-bc64-576cc553de73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ea5e601-k8zn9\" (UID: \"867af907-22c0-4265-bc64-576cc553de73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea5e601-k8zn9" Oct 29 22:52:16.382713 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:16.382594 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szk6n\" (UniqueName: \"kubernetes.io/projected/867af907-22c0-4265-bc64-576cc553de73-kube-api-access-szk6n\") pod \"maas-api-revoke-sub-2ea5e601-k8zn9\" (UID: \"867af907-22c0-4265-bc64-576cc553de73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea5e601-k8zn9" Oct 29 22:52:16.382713 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:16.382654 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867af907-22c0-4265-bc64-576cc553de73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ea5e601-k8zn9\" (UID: \"867af907-22c0-4265-bc64-576cc553de73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea5e601-k8zn9" Oct 29 22:52:16.383291 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:16.383269 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867af907-22c0-4265-bc64-576cc553de73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ea5e601-k8zn9\" (UID: \"867af907-22c0-4265-bc64-576cc553de73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea5e601-k8zn9" Oct 29 22:52:16.385078 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:16.385061 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867af907-22c0-4265-bc64-576cc553de73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ea5e601-k8zn9\" (UID: \"867af907-22c0-4265-bc64-576cc553de73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea5e601-k8zn9" Oct 29 22:52:16.405374 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:16.405346 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szk6n\" (UniqueName: \"kubernetes.io/projected/867af907-22c0-4265-bc64-576cc553de73-kube-api-access-szk6n\") pod \"maas-api-revoke-sub-2ea5e601-k8zn9\" (UID: \"867af907-22c0-4265-bc64-576cc553de73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea5e601-k8zn9" Oct 29 22:52:16.527661 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:16.527592 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea5e601-k8zn9" Oct 29 22:52:16.828604 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:16.828576 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea5e601-k8zn9"] Oct 29 22:52:16.830419 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:52:16.830390 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867af907_22c0_4265_bc64_576cc553de73.slice/crio-8c6ad794e9e9bea77217801aeaad93134f11311916f2a585c198c4337683ac7c WatchSource:0}: Error finding container 8c6ad794e9e9bea77217801aeaad93134f11311916f2a585c198c4337683ac7c: Status 404 returned error can't find the container with id 8c6ad794e9e9bea77217801aeaad93134f11311916f2a585c198c4337683ac7c Oct 29 22:52:17.722820 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:17.722787 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea5e601-k8zn9" event={"ID":"867af907-22c0-4265-bc64-576cc553de73","Type":"ContainerStarted","Data":"8c6ad794e9e9bea77217801aeaad93134f11311916f2a585c198c4337683ac7c"} Oct 29 22:52:19.736969 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:19.736884 4821 generic.go:356] "Generic (PLEG): container finished" podID="867af907-22c0-4265-bc64-576cc553de73" containerID="41f542a9ba547fd9edf6fd0eca81ac2b890495c811db97dd164fdd51fec95ba4" exitCode=0 Oct 29 22:52:19.737368 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:19.736974 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea5e601-k8zn9" event={"ID":"867af907-22c0-4265-bc64-576cc553de73","Type":"ContainerDied","Data":"41f542a9ba547fd9edf6fd0eca81ac2b890495c811db97dd164fdd51fec95ba4"} Oct 29 22:52:20.915964 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:20.915935 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea5e601-k8zn9" Oct 29 22:52:21.042164 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:21.042059 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/867af907-22c0-4265-bc64-576cc553de73-kube-api-access-szk6n\" (UniqueName: \"kubernetes.io/projected/867af907-22c0-4265-bc64-576cc553de73-kube-api-access-szk6n\") pod \"867af907-22c0-4265-bc64-576cc553de73\" (UID: \"867af907-22c0-4265-bc64-576cc553de73\") " Oct 29 22:52:21.042164 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:21.042144 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/867af907-22c0-4265-bc64-576cc553de73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867af907-22c0-4265-bc64-576cc553de73-maas-api-xks-ca\") pod \"867af907-22c0-4265-bc64-576cc553de73\" (UID: \"867af907-22c0-4265-bc64-576cc553de73\") " Oct 29 22:52:21.042467 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:21.042185 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/867af907-22c0-4265-bc64-576cc553de73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867af907-22c0-4265-bc64-576cc553de73-maas-api-openshift-ca\") pod \"867af907-22c0-4265-bc64-576cc553de73\" (UID: \"867af907-22c0-4265-bc64-576cc553de73\") " Oct 29 22:52:21.043148 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:21.043088 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/867af907-22c0-4265-bc64-576cc553de73-maas-api-openshift-ca" pod "867af907-22c0-4265-bc64-576cc553de73" (UID: "867af907-22c0-4265-bc64-576cc553de73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:21.063069 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:21.062879 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/867af907-22c0-4265-bc64-576cc553de73-maas-api-xks-ca" pod "867af907-22c0-4265-bc64-576cc553de73" (UID: "867af907-22c0-4265-bc64-576cc553de73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:21.063518 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:21.063471 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/867af907-22c0-4265-bc64-576cc553de73-kube-api-access-szk6n" pod "867af907-22c0-4265-bc64-576cc553de73" (UID: "867af907-22c0-4265-bc64-576cc553de73"). InnerVolumeSpecName "kube-api-access-szk6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:21.143816 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:21.143777 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867af907-22c0-4265-bc64-576cc553de73-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:52:21.143816 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:21.143818 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szk6n\" (UniqueName: \"kubernetes.io/projected/867af907-22c0-4265-bc64-576cc553de73-kube-api-access-szk6n\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:52:21.144031 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:21.143835 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867af907-22c0-4265-bc64-576cc553de73-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:52:21.787513 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:21.787473 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea5e601-k8zn9" Oct 29 22:52:21.787813 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:21.787504 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea5e601-k8zn9" event={"ID":"867af907-22c0-4265-bc64-576cc553de73","Type":"ContainerDied","Data":"8c6ad794e9e9bea77217801aeaad93134f11311916f2a585c198c4337683ac7c"} Oct 29 22:52:21.787940 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:21.787857 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c6ad794e9e9bea77217801aeaad93134f11311916f2a585c198c4337683ac7c" Oct 29 22:52:23.429144 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:23.429116 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:52:23.442282 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:23.442255 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:52:23.810156 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:23.810063 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:52:23.887118 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:52:23.887079 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:52:37.849248 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:37.849210 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c"] Oct 29 22:52:37.849667 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:37.849580 4821 cpu_manager.go:414] "containerMap: removing container" podUID="867af907-22c0-4265-bc64-576cc553de73" containerName="revoke-keys" Oct 29 22:52:37.849667 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:37.849592 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="867af907-22c0-4265-bc64-576cc553de73" containerName="revoke-keys" Oct 29 22:52:37.849741 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:37.849684 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="867af907-22c0-4265-bc64-576cc553de73" containerName="revoke-keys" Oct 29 22:52:37.854008 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:37.853984 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:37.870492 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:37.870468 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-08e15d-openshift-default-dockercfg-kjhd8\"" Oct 29 22:52:37.885640 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:37.885609 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c"] Oct 29 22:52:37.946772 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:37.946740 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:37.946772 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:37.946774 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:37.946963 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:37.946797 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:37.946963 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:37.946872 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:37.946963 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:37.946892 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:37.946963 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:37.946936 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:37.947089 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:37.947035 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:37.947122 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:37.947095 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:37.947180 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:37.947160 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z9pf\" (UniqueName: \"kubernetes.io/projected/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-kube-api-access-9z9pf\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:38.048166 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.048136 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:38.048655 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.048189 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z9pf\" (UniqueName: \"kubernetes.io/projected/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-kube-api-access-9z9pf\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:38.048655 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.048250 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:38.048655 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.048270 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:38.048655 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.048300 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:38.048655 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.048347 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:38.048655 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.048377 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:38.048655 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.048454 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:38.048655 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.048494 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:38.049602 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.048657 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:38.049602 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.049150 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:38.049602 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.049556 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:38.049602 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.049560 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:38.049748 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.049725 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:38.051270 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.051243 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:38.051388 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.051245 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:38.080730 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.080707 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:38.081084 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.081064 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z9pf\" (UniqueName: \"kubernetes.io/projected/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-kube-api-access-9z9pf\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:38.168017 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.167973 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:38.490727 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.490674 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c"] Oct 29 22:52:38.493173 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:52:38.493145 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c8fc4f_604c_4938_b9e1_5745ac6558c7.slice/crio-38627a23370f3f31a0de0883c561d36e08a8ebf85ceeab8ec683d0a9716afbe3 WatchSource:0}: Error finding container 38627a23370f3f31a0de0883c561d36e08a8ebf85ceeab8ec683d0a9716afbe3: Status 404 returned error can't find the container with id 38627a23370f3f31a0de0883c561d36e08a8ebf85ceeab8ec683d0a9716afbe3 Oct 29 22:52:38.495704 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.495670 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:52:38.495800 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.495737 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:52:38.495800 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.495765 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:52:38.894658 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.894616 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" event={"ID":"d3c8fc4f-604c-4938-b9e1-5745ac6558c7","Type":"ContainerStarted","Data":"2c4d24aae0b2829c32abe6b5ba1c0b2488448ce49fbccb97e38e5d462dd0e2c6"} Oct 29 22:52:38.895147 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.894686 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" event={"ID":"d3c8fc4f-604c-4938-b9e1-5745ac6558c7","Type":"ContainerStarted","Data":"38627a23370f3f31a0de0883c561d36e08a8ebf85ceeab8ec683d0a9716afbe3"} Oct 29 22:52:38.940961 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:38.940887 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" podStartSLOduration=1.940872022 podStartE2EDuration="1.940872022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:38.938303058 +0000 UTC m=+612.878850968" watchObservedRunningTime="2026-10-29 22:52:38.940872022 +0000 UTC m=+612.881419932" Oct 29 22:52:39.169152 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:39.169065 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:39.170841 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:39.170811 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 29 22:52:39.170957 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:39.170866 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" podUID="d3c8fc4f-604c-4938-b9e1-5745ac6558c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 29 22:52:40.147381 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.147351 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx"] Oct 29 22:52:40.151864 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.151836 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:52:40.169239 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.168805 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 29 22:52:40.169239 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.168887 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" podUID="d3c8fc4f-604c-4938-b9e1-5745ac6558c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 29 22:52:40.178651 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.176623 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx"] Oct 29 22:52:40.181374 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.181353 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-063b46\"/\"kube-root-ca.crt\"" Oct 29 22:52:40.181533 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.181393 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-063b46\"/\"default-dockercfg-8h2pz\"" Oct 29 22:52:40.182497 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.182478 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-063b46\"/\"premium-w4-063b46-kserve-self-signed-certs\"" Oct 29 22:52:40.182575 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.182520 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-063b46\"/\"openshift-service-ca.crt\"" Oct 29 22:52:40.282713 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.282672 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-dshm\") pod \"premium-w4-063b46-kserve-6c46d9cfcc-fp7fx\" (UID: \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\") " pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:52:40.282903 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.282820 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-tls-certs\") pod \"premium-w4-063b46-kserve-6c46d9cfcc-fp7fx\" (UID: \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\") " pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:52:40.282903 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.282856 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-model-cache\") pod \"premium-w4-063b46-kserve-6c46d9cfcc-fp7fx\" (UID: \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\") " pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:52:40.282903 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.282886 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-tmp-dir\") pod \"premium-w4-063b46-kserve-6c46d9cfcc-fp7fx\" (UID: \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\") " pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:52:40.283065 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.282973 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7txf\" (UniqueName: \"kubernetes.io/projected/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-kube-api-access-x7txf\") pod \"premium-w4-063b46-kserve-6c46d9cfcc-fp7fx\" (UID: \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\") " pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:52:40.283065 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.283009 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-home\") pod \"premium-w4-063b46-kserve-6c46d9cfcc-fp7fx\" (UID: \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\") " pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:52:40.385640 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.384266 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-dshm\") pod \"premium-w4-063b46-kserve-6c46d9cfcc-fp7fx\" (UID: \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\") " pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:52:40.385640 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.384360 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-tls-certs\") pod \"premium-w4-063b46-kserve-6c46d9cfcc-fp7fx\" (UID: \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\") " pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:52:40.385640 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.384394 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-model-cache\") pod \"premium-w4-063b46-kserve-6c46d9cfcc-fp7fx\" (UID: \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\") " pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:52:40.385640 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.384434 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-tmp-dir\") pod \"premium-w4-063b46-kserve-6c46d9cfcc-fp7fx\" (UID: \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\") " pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:52:40.385640 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.384472 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7txf\" (UniqueName: \"kubernetes.io/projected/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-kube-api-access-x7txf\") pod \"premium-w4-063b46-kserve-6c46d9cfcc-fp7fx\" (UID: \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\") " pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:52:40.385640 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.384503 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-home\") pod \"premium-w4-063b46-kserve-6c46d9cfcc-fp7fx\" (UID: \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\") " pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:52:40.385640 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.385025 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-home\") pod \"premium-w4-063b46-kserve-6c46d9cfcc-fp7fx\" (UID: \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\") " pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:52:40.385640 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.385337 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-model-cache\") pod \"premium-w4-063b46-kserve-6c46d9cfcc-fp7fx\" (UID: \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\") " pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:52:40.387591 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.386915 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-tmp-dir\") pod \"premium-w4-063b46-kserve-6c46d9cfcc-fp7fx\" (UID: \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\") " pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:52:40.390769 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.390273 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-dshm\") pod \"premium-w4-063b46-kserve-6c46d9cfcc-fp7fx\" (UID: \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\") " pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:52:40.390769 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.390724 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-tls-certs\") pod \"premium-w4-063b46-kserve-6c46d9cfcc-fp7fx\" (UID: \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\") " pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:52:40.409600 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.409496 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7txf\" (UniqueName: \"kubernetes.io/projected/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-kube-api-access-x7txf\") pod \"premium-w4-063b46-kserve-6c46d9cfcc-fp7fx\" (UID: \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\") " pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:52:40.466445 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.466407 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:52:40.803595 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.803570 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx"] Oct 29 22:52:40.804913 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:52:40.804878 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc436ecf5_3fdf_43ea_be7e_d58eaf7481b6.slice/crio-349992c5575f141639db54f7d2a0aff701112953a6e464ae0efc8772fec65a60 WatchSource:0}: Error finding container 349992c5575f141639db54f7d2a0aff701112953a6e464ae0efc8772fec65a60: Status 404 returned error can't find the container with id 349992c5575f141639db54f7d2a0aff701112953a6e464ae0efc8772fec65a60 Oct 29 22:52:40.908664 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.908630 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" event={"ID":"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6","Type":"ContainerStarted","Data":"91b87b818340ad5872492d15dd0124d96523708b9e2b15d866295e13b6f1df4e"} Oct 29 22:52:40.908899 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.908866 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" event={"ID":"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6","Type":"ContainerStarted","Data":"349992c5575f141639db54f7d2a0aff701112953a6e464ae0efc8772fec65a60"} Oct 29 22:52:40.952948 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:40.952865 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podStartSLOduration=0.952840806 podStartE2EDuration="952.840806ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:40.949576226 +0000 UTC m=+614.890124136" watchObservedRunningTime="2026-10-29 22:52:40.952840806 +0000 UTC m=+614.893388717" Oct 29 22:52:41.168899 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:41.168864 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 29 22:52:41.169326 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:41.168921 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" podUID="d3c8fc4f-604c-4938-b9e1-5745ac6558c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 29 22:52:41.466679 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:41.466585 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:52:42.168692 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:42.168659 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 29 22:52:42.168862 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:42.168707 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" podUID="d3c8fc4f-604c-4938-b9e1-5745ac6558c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 29 22:52:43.169052 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:43.169022 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 29 22:52:43.169472 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:43.169069 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" podUID="d3c8fc4f-604c-4938-b9e1-5745ac6558c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 29 22:52:44.169162 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.169129 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 29 22:52:44.169665 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.169183 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" podUID="d3c8fc4f-604c-4938-b9e1-5745ac6558c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 29 22:52:44.515460 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.515384 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4"] Oct 29 22:52:44.519067 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.519044 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.541004 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.540978 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.541156 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.541008 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.541156 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.541028 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.541287 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.541188 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.541287 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.541275 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.541369 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.541307 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.541369 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.541339 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.541440 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.541374 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngxpk\" (UniqueName: \"kubernetes.io/projected/9be68e75-df62-4a43-9ae9-20c04683f4c7-kube-api-access-ngxpk\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.541487 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.541472 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9be68e75-df62-4a43-9ae9-20c04683f4c7-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.552109 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.552077 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4"] Oct 29 22:52:44.642793 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.642754 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.642965 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.642798 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.642965 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.642828 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.642965 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.642894 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.642965 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.642948 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.643263 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.642986 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.643263 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.643045 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.643263 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.643113 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngxpk\" (UniqueName: \"kubernetes.io/projected/9be68e75-df62-4a43-9ae9-20c04683f4c7-kube-api-access-ngxpk\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.643263 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.643173 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9be68e75-df62-4a43-9ae9-20c04683f4c7-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.643263 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.643252 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.643544 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.643386 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.643544 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.643423 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.643645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.643546 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.644059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.644009 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9be68e75-df62-4a43-9ae9-20c04683f4c7-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.645595 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.645571 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.645889 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.645865 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.665573 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.665546 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngxpk\" (UniqueName: \"kubernetes.io/projected/9be68e75-df62-4a43-9ae9-20c04683f4c7-kube-api-access-ngxpk\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.665573 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.665555 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:44.839423 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:44.839334 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:45.185508 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:45.185146 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4"] Oct 29 22:52:45.187601 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:52:45.187569 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be68e75_df62_4a43_9ae9_20c04683f4c7.slice/crio-533b4ba493162a6275126c18e024be067a7b3986e9b2de7241d9f57100229c2e WatchSource:0}: Error finding container 533b4ba493162a6275126c18e024be067a7b3986e9b2de7241d9f57100229c2e: Status 404 returned error can't find the container with id 533b4ba493162a6275126c18e024be067a7b3986e9b2de7241d9f57100229c2e Oct 29 22:52:45.190456 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:45.190418 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:52:45.190539 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:45.190505 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:52:45.190578 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:45.190547 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:52:45.212025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:45.212001 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:45.212475 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:45.212415 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:45.214439 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:45.214417 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 22:52:45.963126 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:45.963085 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" event={"ID":"9be68e75-df62-4a43-9ae9-20c04683f4c7","Type":"ContainerStarted","Data":"7934b7a54f73d2c8220c99250e75365b6e5c194a793de9b3ca9125e554500004"} Oct 29 22:52:45.963126 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:45.963132 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" event={"ID":"9be68e75-df62-4a43-9ae9-20c04683f4c7","Type":"ContainerStarted","Data":"533b4ba493162a6275126c18e024be067a7b3986e9b2de7241d9f57100229c2e"} Oct 29 22:52:46.842139 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:46.842092 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 29 22:52:46.842658 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:46.842375 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" podUID="9be68e75-df62-4a43-9ae9-20c04683f4c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 29 22:52:46.842658 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:46.842293 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:47.841662 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:47.841623 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 29 22:52:47.841991 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:47.841942 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" podUID="9be68e75-df62-4a43-9ae9-20c04683f4c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 29 22:52:48.840602 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:48.840558 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 29 22:52:48.841008 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:48.840636 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" podUID="9be68e75-df62-4a43-9ae9-20c04683f4c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 29 22:52:49.843224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:49.842592 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 29 22:52:49.843224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:49.842668 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" podUID="9be68e75-df62-4a43-9ae9-20c04683f4c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 29 22:52:50.467101 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:50.467064 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:52:50.480075 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:50.480049 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:52:50.524227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:50.523845 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" podStartSLOduration=6.52382607 podStartE2EDuration="6.52382607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:46.023318983 +0000 UTC m=+619.963866896" watchObservedRunningTime="2026-10-29 22:52:50.52382607 +0000 UTC m=+624.464373982" Oct 29 22:52:50.840744 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:50.840651 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 29 22:52:50.840744 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:50.840724 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" podUID="9be68e75-df62-4a43-9ae9-20c04683f4c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 29 22:52:51.003047 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:51.003011 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:52:52.482363 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:52.482329 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:52.482885 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:52.482807 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:52.484341 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:52.484318 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 22:52:53.883671 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:52:53.883629 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:52:56.420659 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:56.420624 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg"] Oct 29 22:52:56.428435 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:56.428410 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:52:56.448191 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:56.448151 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg"] Oct 29 22:52:56.455330 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:56.455307 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e131af\"/\"e2e-conflict-a-2f81a573-kserve-self-signed-certs\"" Oct 29 22:52:56.482333 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:56.482298 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-dshm\") pod \"e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg\" (UID: \"18e49082-9069-46f6-ba3c-409196298ce4\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:52:56.482552 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:56.482364 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-model-cache\") pod \"e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg\" (UID: \"18e49082-9069-46f6-ba3c-409196298ce4\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:52:56.482552 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:56.482491 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-home\") pod \"e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg\" (UID: \"18e49082-9069-46f6-ba3c-409196298ce4\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:52:56.482552 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:56.482540 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e49082-9069-46f6-ba3c-409196298ce4-tls-certs\") pod \"e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg\" (UID: \"18e49082-9069-46f6-ba3c-409196298ce4\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:52:56.482735 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:56.482568 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-tmp-dir\") pod \"e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg\" (UID: \"18e49082-9069-46f6-ba3c-409196298ce4\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:52:56.482735 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:56.482594 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44c9x\" (UniqueName: \"kubernetes.io/projected/18e49082-9069-46f6-ba3c-409196298ce4-kube-api-access-44c9x\") pod \"e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg\" (UID: \"18e49082-9069-46f6-ba3c-409196298ce4\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:52:56.583591 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:56.583547 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-dshm\") pod \"e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg\" (UID: \"18e49082-9069-46f6-ba3c-409196298ce4\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:52:56.583791 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:56.583616 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-model-cache\") pod \"e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg\" (UID: \"18e49082-9069-46f6-ba3c-409196298ce4\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:52:56.583791 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:56.583688 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-home\") pod \"e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg\" (UID: \"18e49082-9069-46f6-ba3c-409196298ce4\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:52:56.583791 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:56.583748 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e49082-9069-46f6-ba3c-409196298ce4-tls-certs\") pod \"e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg\" (UID: \"18e49082-9069-46f6-ba3c-409196298ce4\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:52:56.583964 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:56.583791 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-tmp-dir\") pod \"e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg\" (UID: \"18e49082-9069-46f6-ba3c-409196298ce4\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:52:56.583964 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:56.583831 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44c9x\" (UniqueName: \"kubernetes.io/projected/18e49082-9069-46f6-ba3c-409196298ce4-kube-api-access-44c9x\") pod \"e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg\" (UID: \"18e49082-9069-46f6-ba3c-409196298ce4\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:52:56.584395 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:56.584035 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-model-cache\") pod \"e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg\" (UID: \"18e49082-9069-46f6-ba3c-409196298ce4\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:52:56.584395 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:56.584366 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-home\") pod \"e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg\" (UID: \"18e49082-9069-46f6-ba3c-409196298ce4\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:52:56.584546 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:56.584437 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-tmp-dir\") pod \"e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg\" (UID: \"18e49082-9069-46f6-ba3c-409196298ce4\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:52:56.586503 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:56.586470 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e49082-9069-46f6-ba3c-409196298ce4-tls-certs\") pod \"e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg\" (UID: \"18e49082-9069-46f6-ba3c-409196298ce4\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:52:56.586503 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:56.586498 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-dshm\") pod \"e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg\" (UID: \"18e49082-9069-46f6-ba3c-409196298ce4\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:52:56.607603 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:56.607568 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44c9x\" (UniqueName: \"kubernetes.io/projected/18e49082-9069-46f6-ba3c-409196298ce4-kube-api-access-44c9x\") pod \"e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg\" (UID: \"18e49082-9069-46f6-ba3c-409196298ce4\") " pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:52:56.742758 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:56.742672 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:52:57.050731 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:57.050650 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg"] Oct 29 22:52:57.054830 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:52:57.054800 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18e49082_9069_46f6_ba3c_409196298ce4.slice/crio-6369901ad70feb9ef1158f967eadbc8feb01b50cd215973bdcebdcb58d4267c2 WatchSource:0}: Error finding container 6369901ad70feb9ef1158f967eadbc8feb01b50cd215973bdcebdcb58d4267c2: Status 404 returned error can't find the container with id 6369901ad70feb9ef1158f967eadbc8feb01b50cd215973bdcebdcb58d4267c2 Oct 29 22:52:58.054467 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:58.054413 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" event={"ID":"18e49082-9069-46f6-ba3c-409196298ce4","Type":"ContainerStarted","Data":"32d1a4322eb69b25bb7bc85e1759151c1b9af40232eaeb06d6de50675d514bc1"} Oct 29 22:52:58.055091 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:58.055044 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" event={"ID":"18e49082-9069-46f6-ba3c-409196298ce4","Type":"ContainerStarted","Data":"6369901ad70feb9ef1158f967eadbc8feb01b50cd215973bdcebdcb58d4267c2"} Oct 29 22:52:58.102235 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:58.101966 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" podStartSLOduration=2.101945715 podStartE2EDuration="2.101945715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:58.099660376 +0000 UTC m=+632.040208287" watchObservedRunningTime="2026-10-29 22:52:58.101945715 +0000 UTC m=+632.042493627" Oct 29 22:52:58.743367 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:52:58.743333 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:53:06.743018 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:06.742976 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:53:06.756097 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:06.756072 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:53:07.119335 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:07.119306 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:53:08.840224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:08.839960 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw"] Oct 29 22:53:08.846244 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:08.844562 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:08.888133 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:08.866609 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-17394a-openshift-default-dockercfg-x6t7f\"" Oct 29 22:53:08.897284 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:08.897254 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw"] Oct 29 22:53:08.928277 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:08.928003 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-workload-certs\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:08.928277 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:08.928192 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c103357b-acd8-44e6-9600-c56965634907-istio-podinfo\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:08.931958 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:08.928345 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vfdd\" (UniqueName: \"kubernetes.io/projected/c103357b-acd8-44e6-9600-c56965634907-kube-api-access-8vfdd\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:08.931958 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:08.928390 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-credential-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:08.931958 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:08.928431 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c103357b-acd8-44e6-9600-c56965634907-istio-token\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:08.931958 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:08.928466 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c103357b-acd8-44e6-9600-c56965634907-istiod-ca-cert\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:08.931958 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:08.928520 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-workload-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:08.931958 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:08.928562 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-istio-envoy\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:08.931958 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:08.928593 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-istio-data\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:09.029822 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:09.029778 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c103357b-acd8-44e6-9600-c56965634907-istio-token\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:09.032571 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:09.030063 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c103357b-acd8-44e6-9600-c56965634907-istiod-ca-cert\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:09.032571 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:09.030129 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-workload-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:09.032571 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:09.030175 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-istio-envoy\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:09.032571 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:09.030234 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-istio-data\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:09.032571 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:09.030307 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-workload-certs\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:09.032571 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:09.030413 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c103357b-acd8-44e6-9600-c56965634907-istio-podinfo\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:09.032571 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:09.030471 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vfdd\" (UniqueName: \"kubernetes.io/projected/c103357b-acd8-44e6-9600-c56965634907-kube-api-access-8vfdd\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:09.032571 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:09.030508 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-credential-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:09.032571 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:09.030959 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-credential-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:09.032571 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:09.031965 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c103357b-acd8-44e6-9600-c56965634907-istiod-ca-cert\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:09.032571 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:09.032249 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-workload-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:09.033614 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:09.033591 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-workload-certs\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:09.033992 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:09.033970 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-istio-data\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:09.047783 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:09.047759 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c103357b-acd8-44e6-9600-c56965634907-istio-podinfo\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:09.084241 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:09.068344 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-istio-envoy\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:09.084440 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:09.075019 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vfdd\" (UniqueName: \"kubernetes.io/projected/c103357b-acd8-44e6-9600-c56965634907-kube-api-access-8vfdd\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:09.084572 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:09.075683 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c103357b-acd8-44e6-9600-c56965634907-istio-token\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:09.209330 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:09.209286 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:09.583812 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:09.583787 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw"] Oct 29 22:53:09.586190 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:53:09.586164 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc103357b_acd8_44e6_9600_c56965634907.slice/crio-0905d89c49df2ed52a1db101d9dc4f5981a6cab39c5414043e1247f9706141f6 WatchSource:0}: Error finding container 0905d89c49df2ed52a1db101d9dc4f5981a6cab39c5414043e1247f9706141f6: Status 404 returned error can't find the container with id 0905d89c49df2ed52a1db101d9dc4f5981a6cab39c5414043e1247f9706141f6 Oct 29 22:53:09.588538 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:09.588509 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:53:09.588620 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:09.588575 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:53:09.588620 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:09.588605 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:53:10.143803 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:10.143767 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" event={"ID":"c103357b-acd8-44e6-9600-c56965634907","Type":"ContainerStarted","Data":"6f6835c53e154f1a4f6b04702bc55223cf26d5db328734a4cebd949b9cbf9be6"} Oct 29 22:53:10.144240 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:10.143849 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" event={"ID":"c103357b-acd8-44e6-9600-c56965634907","Type":"ContainerStarted","Data":"0905d89c49df2ed52a1db101d9dc4f5981a6cab39c5414043e1247f9706141f6"} Oct 29 22:53:10.200821 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:10.200768 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" podStartSLOduration=2.200755084 podStartE2EDuration="2.200755084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:10.187059865 +0000 UTC m=+644.127607776" watchObservedRunningTime="2026-10-29 22:53:10.200755084 +0000 UTC m=+644.141302993" Oct 29 22:53:10.209635 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:10.209605 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:10.211167 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:10.211141 4821 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 29 22:53:10.211299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:10.211183 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" podUID="c103357b-acd8-44e6-9600-c56965634907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 29 22:53:11.209660 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:11.209625 4821 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 29 22:53:11.210134 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:11.209692 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" podUID="c103357b-acd8-44e6-9600-c56965634907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 29 22:53:12.209829 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:12.209794 4821 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 29 22:53:12.210276 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:12.209861 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" podUID="c103357b-acd8-44e6-9600-c56965634907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 29 22:53:13.210429 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:13.210381 4821 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 29 22:53:13.210829 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:13.210452 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" podUID="c103357b-acd8-44e6-9600-c56965634907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 29 22:53:14.209904 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:14.209863 4821 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 29 22:53:14.210102 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:14.209935 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" podUID="c103357b-acd8-44e6-9600-c56965634907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 29 22:53:15.210125 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:15.210088 4821 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 29 22:53:15.210657 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:15.210167 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" podUID="c103357b-acd8-44e6-9600-c56965634907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 29 22:53:16.210368 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:16.210332 4821 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 29 22:53:16.210881 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:16.210383 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" podUID="c103357b-acd8-44e6-9600-c56965634907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 29 22:53:17.215167 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:17.215131 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:17.215698 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:17.215677 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:17.216480 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:17.216462 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 22:53:23.892149 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:53:23.892107 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:53:33.544631 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:33.544593 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7"] Oct 29 22:53:33.550571 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:33.550547 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:53:33.570235 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:33.570178 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7"] Oct 29 22:53:33.583525 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:33.583461 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-17394a\"/\"distinct-w3-17394a-kserve-self-signed-certs\"" Oct 29 22:53:33.642938 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:33.642899 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-tmp-dir\") pod \"distinct-w3-17394a-kserve-57c47678df-wtmr7\" (UID: \"1231f78b-aaa2-4595-a11e-fcdbf984521e\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:53:33.643120 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:33.642974 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cszm2\" (UniqueName: \"kubernetes.io/projected/1231f78b-aaa2-4595-a11e-fcdbf984521e-kube-api-access-cszm2\") pod \"distinct-w3-17394a-kserve-57c47678df-wtmr7\" (UID: \"1231f78b-aaa2-4595-a11e-fcdbf984521e\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:53:33.643120 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:33.643072 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1231f78b-aaa2-4595-a11e-fcdbf984521e-tls-certs\") pod \"distinct-w3-17394a-kserve-57c47678df-wtmr7\" (UID: \"1231f78b-aaa2-4595-a11e-fcdbf984521e\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:53:33.643230 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:33.643163 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-model-cache\") pod \"distinct-w3-17394a-kserve-57c47678df-wtmr7\" (UID: \"1231f78b-aaa2-4595-a11e-fcdbf984521e\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:53:33.643297 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:33.643276 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-home\") pod \"distinct-w3-17394a-kserve-57c47678df-wtmr7\" (UID: \"1231f78b-aaa2-4595-a11e-fcdbf984521e\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:53:33.643333 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:33.643324 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-dshm\") pod \"distinct-w3-17394a-kserve-57c47678df-wtmr7\" (UID: \"1231f78b-aaa2-4595-a11e-fcdbf984521e\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:53:33.749022 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:33.747862 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cszm2\" (UniqueName: \"kubernetes.io/projected/1231f78b-aaa2-4595-a11e-fcdbf984521e-kube-api-access-cszm2\") pod \"distinct-w3-17394a-kserve-57c47678df-wtmr7\" (UID: \"1231f78b-aaa2-4595-a11e-fcdbf984521e\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:53:33.749022 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:33.747922 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1231f78b-aaa2-4595-a11e-fcdbf984521e-tls-certs\") pod \"distinct-w3-17394a-kserve-57c47678df-wtmr7\" (UID: \"1231f78b-aaa2-4595-a11e-fcdbf984521e\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:53:33.749022 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:33.747997 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-model-cache\") pod \"distinct-w3-17394a-kserve-57c47678df-wtmr7\" (UID: \"1231f78b-aaa2-4595-a11e-fcdbf984521e\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:53:33.749022 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:33.748095 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-home\") pod \"distinct-w3-17394a-kserve-57c47678df-wtmr7\" (UID: \"1231f78b-aaa2-4595-a11e-fcdbf984521e\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:53:33.749022 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:33.748156 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-dshm\") pod \"distinct-w3-17394a-kserve-57c47678df-wtmr7\" (UID: \"1231f78b-aaa2-4595-a11e-fcdbf984521e\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:53:33.749022 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:33.748250 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-tmp-dir\") pod \"distinct-w3-17394a-kserve-57c47678df-wtmr7\" (UID: \"1231f78b-aaa2-4595-a11e-fcdbf984521e\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:53:33.749022 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:33.748814 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-tmp-dir\") pod \"distinct-w3-17394a-kserve-57c47678df-wtmr7\" (UID: \"1231f78b-aaa2-4595-a11e-fcdbf984521e\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:53:33.755267 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:33.750010 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-home\") pod \"distinct-w3-17394a-kserve-57c47678df-wtmr7\" (UID: \"1231f78b-aaa2-4595-a11e-fcdbf984521e\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:53:33.755267 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:33.750342 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-model-cache\") pod \"distinct-w3-17394a-kserve-57c47678df-wtmr7\" (UID: \"1231f78b-aaa2-4595-a11e-fcdbf984521e\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:53:33.755267 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:33.753710 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1231f78b-aaa2-4595-a11e-fcdbf984521e-tls-certs\") pod \"distinct-w3-17394a-kserve-57c47678df-wtmr7\" (UID: \"1231f78b-aaa2-4595-a11e-fcdbf984521e\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:53:33.759252 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:33.756534 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-dshm\") pod \"distinct-w3-17394a-kserve-57c47678df-wtmr7\" (UID: \"1231f78b-aaa2-4595-a11e-fcdbf984521e\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:53:33.774120 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:33.774057 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cszm2\" (UniqueName: \"kubernetes.io/projected/1231f78b-aaa2-4595-a11e-fcdbf984521e-kube-api-access-cszm2\") pod \"distinct-w3-17394a-kserve-57c47678df-wtmr7\" (UID: \"1231f78b-aaa2-4595-a11e-fcdbf984521e\") " pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:53:33.863583 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:33.863545 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:53:34.182140 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:34.182106 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7"] Oct 29 22:53:34.183595 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:53:34.183570 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1231f78b_aaa2_4595_a11e_fcdbf984521e.slice/crio-e8e49855510abd06def371bf2e9be525b4875ddc4817f4d7282310b5ce1a6bc4 WatchSource:0}: Error finding container e8e49855510abd06def371bf2e9be525b4875ddc4817f4d7282310b5ce1a6bc4: Status 404 returned error can't find the container with id e8e49855510abd06def371bf2e9be525b4875ddc4817f4d7282310b5ce1a6bc4 Oct 29 22:53:34.306570 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:34.306530 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" event={"ID":"1231f78b-aaa2-4595-a11e-fcdbf984521e","Type":"ContainerStarted","Data":"6098d9779a9f49651f7131fbe7ced8656360875db4c1c11387a3d30af7175bce"} Oct 29 22:53:34.306570 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:34.306576 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" event={"ID":"1231f78b-aaa2-4595-a11e-fcdbf984521e","Type":"ContainerStarted","Data":"e8e49855510abd06def371bf2e9be525b4875ddc4817f4d7282310b5ce1a6bc4"} Oct 29 22:53:34.348303 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:34.348251 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podStartSLOduration=1.348232652 podStartE2EDuration="1.348232652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:34.345806141 +0000 UTC m=+668.286354049" watchObservedRunningTime="2026-10-29 22:53:34.348232652 +0000 UTC m=+668.288780563" Oct 29 22:53:34.864457 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:34.864397 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:53:42.505439 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:42.505353 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr"] Oct 29 22:53:42.509685 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:42.509661 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:53:42.527523 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:42.527487 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b2921b\"/\"distinct-w5-b2921b-kserve-self-signed-certs\"" Oct 29 22:53:42.542587 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:42.542561 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr"] Oct 29 22:53:42.585083 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:42.585047 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-tmp-dir\") pod \"distinct-w5-b2921b-kserve-5d966f7484-2c4lr\" (UID: \"9ca8086c-12ec-4fc1-b914-607bd9789748\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:53:42.585269 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:42.585168 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-home\") pod \"distinct-w5-b2921b-kserve-5d966f7484-2c4lr\" (UID: \"9ca8086c-12ec-4fc1-b914-607bd9789748\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:53:42.585269 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:42.585261 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca8086c-12ec-4fc1-b914-607bd9789748-tls-certs\") pod \"distinct-w5-b2921b-kserve-5d966f7484-2c4lr\" (UID: \"9ca8086c-12ec-4fc1-b914-607bd9789748\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:53:42.585383 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:42.585295 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-model-cache\") pod \"distinct-w5-b2921b-kserve-5d966f7484-2c4lr\" (UID: \"9ca8086c-12ec-4fc1-b914-607bd9789748\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:53:42.585486 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:42.585465 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z4mm\" (UniqueName: \"kubernetes.io/projected/9ca8086c-12ec-4fc1-b914-607bd9789748-kube-api-access-8z4mm\") pod \"distinct-w5-b2921b-kserve-5d966f7484-2c4lr\" (UID: \"9ca8086c-12ec-4fc1-b914-607bd9789748\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:53:42.585560 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:42.585542 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-dshm\") pod \"distinct-w5-b2921b-kserve-5d966f7484-2c4lr\" (UID: \"9ca8086c-12ec-4fc1-b914-607bd9789748\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:53:42.686496 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:42.686458 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-tmp-dir\") pod \"distinct-w5-b2921b-kserve-5d966f7484-2c4lr\" (UID: \"9ca8086c-12ec-4fc1-b914-607bd9789748\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:53:42.686701 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:42.686542 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-home\") pod \"distinct-w5-b2921b-kserve-5d966f7484-2c4lr\" (UID: \"9ca8086c-12ec-4fc1-b914-607bd9789748\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:53:42.686701 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:42.686584 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca8086c-12ec-4fc1-b914-607bd9789748-tls-certs\") pod \"distinct-w5-b2921b-kserve-5d966f7484-2c4lr\" (UID: \"9ca8086c-12ec-4fc1-b914-607bd9789748\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:53:42.686701 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:42.686616 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-model-cache\") pod \"distinct-w5-b2921b-kserve-5d966f7484-2c4lr\" (UID: \"9ca8086c-12ec-4fc1-b914-607bd9789748\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:53:42.686701 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:42.686697 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z4mm\" (UniqueName: \"kubernetes.io/projected/9ca8086c-12ec-4fc1-b914-607bd9789748-kube-api-access-8z4mm\") pod \"distinct-w5-b2921b-kserve-5d966f7484-2c4lr\" (UID: \"9ca8086c-12ec-4fc1-b914-607bd9789748\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:53:42.686918 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:42.686768 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-dshm\") pod \"distinct-w5-b2921b-kserve-5d966f7484-2c4lr\" (UID: \"9ca8086c-12ec-4fc1-b914-607bd9789748\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:53:42.686965 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:42.686924 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-tmp-dir\") pod \"distinct-w5-b2921b-kserve-5d966f7484-2c4lr\" (UID: \"9ca8086c-12ec-4fc1-b914-607bd9789748\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:53:42.687003 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:42.686962 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-home\") pod \"distinct-w5-b2921b-kserve-5d966f7484-2c4lr\" (UID: \"9ca8086c-12ec-4fc1-b914-607bd9789748\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:53:42.687251 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:42.687226 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-model-cache\") pod \"distinct-w5-b2921b-kserve-5d966f7484-2c4lr\" (UID: \"9ca8086c-12ec-4fc1-b914-607bd9789748\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:53:42.689427 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:42.689402 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-dshm\") pod \"distinct-w5-b2921b-kserve-5d966f7484-2c4lr\" (UID: \"9ca8086c-12ec-4fc1-b914-607bd9789748\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:53:42.689539 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:42.689526 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca8086c-12ec-4fc1-b914-607bd9789748-tls-certs\") pod \"distinct-w5-b2921b-kserve-5d966f7484-2c4lr\" (UID: \"9ca8086c-12ec-4fc1-b914-607bd9789748\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:53:42.720256 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:42.714891 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z4mm\" (UniqueName: \"kubernetes.io/projected/9ca8086c-12ec-4fc1-b914-607bd9789748-kube-api-access-8z4mm\") pod \"distinct-w5-b2921b-kserve-5d966f7484-2c4lr\" (UID: \"9ca8086c-12ec-4fc1-b914-607bd9789748\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:53:42.824256 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:42.824155 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:53:43.131484 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:43.131453 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr"] Oct 29 22:53:43.132636 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:53:43.132613 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ca8086c_12ec_4fc1_b914_607bd9789748.slice/crio-6dc36a2d48a91f85adb9cca96a15295cea3626ae30cb9cfee19df073b72e5ce4 WatchSource:0}: Error finding container 6dc36a2d48a91f85adb9cca96a15295cea3626ae30cb9cfee19df073b72e5ce4: Status 404 returned error can't find the container with id 6dc36a2d48a91f85adb9cca96a15295cea3626ae30cb9cfee19df073b72e5ce4 Oct 29 22:53:43.386954 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:43.386862 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" event={"ID":"9ca8086c-12ec-4fc1-b914-607bd9789748","Type":"ContainerStarted","Data":"2012d9f90b11484521e675d5e5a701eb71f824e4fdece36bd82207118439ef5e"} Oct 29 22:53:43.386954 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:43.386908 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" event={"ID":"9ca8086c-12ec-4fc1-b914-607bd9789748","Type":"ContainerStarted","Data":"6dc36a2d48a91f85adb9cca96a15295cea3626ae30cb9cfee19df073b72e5ce4"} Oct 29 22:53:43.429614 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:43.429558 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podStartSLOduration=1.429545097 podStartE2EDuration="1.429545097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:43.426239924 +0000 UTC m=+677.366787834" watchObservedRunningTime="2026-10-29 22:53:43.429545097 +0000 UTC m=+677.370093007" Oct 29 22:53:43.824729 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:43.824648 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:53:43.863656 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:43.863627 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:53:43.876739 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:43.876717 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:53:44.404141 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:44.404114 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:53:51.222894 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.222859 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7"] Oct 29 22:53:51.227285 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.227264 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.241834 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.241796 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-063b46-openshift-default-dockercfg-hj7tc\"" Oct 29 22:53:51.256215 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.256180 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7"] Oct 29 22:53:51.298399 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.298364 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.298598 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.298546 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.298598 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.298589 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.298803 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.298780 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmppg\" (UniqueName: \"kubernetes.io/projected/5010e064-9f9b-4de0-8b7b-4c132c3f5026-kube-api-access-hmppg\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.298897 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.298825 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.298962 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.298909 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.299023 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.299005 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.299066 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.299033 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.299246 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.299215 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.399950 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.399914 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.400145 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.400034 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.400145 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.400067 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.400145 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.400125 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmppg\" (UniqueName: \"kubernetes.io/projected/5010e064-9f9b-4de0-8b7b-4c132c3f5026-kube-api-access-hmppg\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.400351 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.400150 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.400351 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.400181 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.400351 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.400251 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.400351 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.400277 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.400351 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.400341 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.400604 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.400550 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.400665 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.400617 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.400864 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.400811 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.400940 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.400861 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.400940 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.400907 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.402784 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.402755 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.402898 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.402862 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.425596 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.425574 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.425775 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.425757 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmppg\" (UniqueName: \"kubernetes.io/projected/5010e064-9f9b-4de0-8b7b-4c132c3f5026-kube-api-access-hmppg\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.543287 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.541031 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:51.866157 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.866130 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7"] Oct 29 22:53:51.867504 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:53:51.867470 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5010e064_9f9b_4de0_8b7b_4c132c3f5026.slice/crio-e97b2cba4e93c50edb1a5283d9da052cc99cbe6064e0deff4a1354de8376a9cd WatchSource:0}: Error finding container e97b2cba4e93c50edb1a5283d9da052cc99cbe6064e0deff4a1354de8376a9cd: Status 404 returned error can't find the container with id e97b2cba4e93c50edb1a5283d9da052cc99cbe6064e0deff4a1354de8376a9cd Oct 29 22:53:51.870022 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.869992 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:53:51.870105 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.870055 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:53:51.870105 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:51.870085 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:53:52.439897 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:52.439859 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" event={"ID":"5010e064-9f9b-4de0-8b7b-4c132c3f5026","Type":"ContainerStarted","Data":"4b31acace189057cb226559e2c87f84d1a6f46983a9dbce039d28fdc0d742de2"} Oct 29 22:53:52.440426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:52.439916 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" event={"ID":"5010e064-9f9b-4de0-8b7b-4c132c3f5026","Type":"ContainerStarted","Data":"e97b2cba4e93c50edb1a5283d9da052cc99cbe6064e0deff4a1354de8376a9cd"} Oct 29 22:53:52.498325 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:52.498271 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" podStartSLOduration=1.498255682 podStartE2EDuration="1.498255682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:52.495620998 +0000 UTC m=+686.436168908" watchObservedRunningTime="2026-10-29 22:53:52.498255682 +0000 UTC m=+686.438803594" Oct 29 22:53:52.541164 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:52.541128 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:52.542687 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:52.542653 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:53:52.542821 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:52.542709 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" podUID="5010e064-9f9b-4de0-8b7b-4c132c3f5026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 29 22:53:52.825141 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:52.825046 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:53:52.838612 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:52.838582 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:53:53.469316 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:53.469288 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:53:53.555228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:53.554046 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:53:53.555228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:53.554111 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" podUID="5010e064-9f9b-4de0-8b7b-4c132c3f5026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 29 22:53:53.884395 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:53:53.884357 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:53:54.542271 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:54.542235 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:53:54.542689 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:54.542299 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" podUID="5010e064-9f9b-4de0-8b7b-4c132c3f5026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 29 22:53:55.542340 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:55.542305 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:53:55.542788 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:55.542365 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" podUID="5010e064-9f9b-4de0-8b7b-4c132c3f5026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 29 22:53:56.543589 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:56.543550 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:53:56.544221 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:56.544169 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" podUID="5010e064-9f9b-4de0-8b7b-4c132c3f5026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 29 22:53:57.569222 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:57.568571 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:53:57.569222 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:57.568684 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" podUID="5010e064-9f9b-4de0-8b7b-4c132c3f5026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 29 22:53:58.542221 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:58.542170 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:53:58.542430 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:58.542258 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" podUID="5010e064-9f9b-4de0-8b7b-4c132c3f5026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 29 22:53:59.561856 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:59.561821 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:59.562430 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:59.562389 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:53:59.564192 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:53:59.564165 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 22:54:08.847337 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:08.847298 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm"] Oct 29 22:54:08.892881 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:08.892837 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:54:08.893825 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:08.892674 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm"] Oct 29 22:54:08.908268 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:08.907630 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-063b46\"/\"distinct-2-w4-063b46-kserve-self-signed-certs\"" Oct 29 22:54:08.987258 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:08.987193 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-model-cache\") pod \"distinct-2-w4-063b46-kserve-7f485fc759-b65nm\" (UID: \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:54:08.987447 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:08.987276 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-dshm\") pod \"distinct-2-w4-063b46-kserve-7f485fc759-b65nm\" (UID: \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:54:08.987447 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:08.987370 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-tmp-dir\") pod \"distinct-2-w4-063b46-kserve-7f485fc759-b65nm\" (UID: \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:54:08.987447 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:08.987420 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e04ee6b-8e5b-4dc1-b001-184689f651e5-tls-certs\") pod \"distinct-2-w4-063b46-kserve-7f485fc759-b65nm\" (UID: \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:54:08.987633 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:08.987463 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-home\") pod \"distinct-2-w4-063b46-kserve-7f485fc759-b65nm\" (UID: \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:54:08.987633 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:08.987519 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvv7c\" (UniqueName: \"kubernetes.io/projected/8e04ee6b-8e5b-4dc1-b001-184689f651e5-kube-api-access-bvv7c\") pod \"distinct-2-w4-063b46-kserve-7f485fc759-b65nm\" (UID: \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:54:09.088767 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:09.088719 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-tmp-dir\") pod \"distinct-2-w4-063b46-kserve-7f485fc759-b65nm\" (UID: \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:54:09.090189 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:09.090154 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-tmp-dir\") pod \"distinct-2-w4-063b46-kserve-7f485fc759-b65nm\" (UID: \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:54:09.090482 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:09.090376 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e04ee6b-8e5b-4dc1-b001-184689f651e5-tls-certs\") pod \"distinct-2-w4-063b46-kserve-7f485fc759-b65nm\" (UID: \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:54:09.090729 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:09.090710 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-home\") pod \"distinct-2-w4-063b46-kserve-7f485fc759-b65nm\" (UID: \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:54:09.091678 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:09.091250 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvv7c\" (UniqueName: \"kubernetes.io/projected/8e04ee6b-8e5b-4dc1-b001-184689f651e5-kube-api-access-bvv7c\") pod \"distinct-2-w4-063b46-kserve-7f485fc759-b65nm\" (UID: \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:54:09.091678 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:09.091328 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-model-cache\") pod \"distinct-2-w4-063b46-kserve-7f485fc759-b65nm\" (UID: \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:54:09.091678 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:09.091368 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-dshm\") pod \"distinct-2-w4-063b46-kserve-7f485fc759-b65nm\" (UID: \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:54:09.091678 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:09.091639 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-model-cache\") pod \"distinct-2-w4-063b46-kserve-7f485fc759-b65nm\" (UID: \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:54:09.091935 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:09.091881 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-home\") pod \"distinct-2-w4-063b46-kserve-7f485fc759-b65nm\" (UID: \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:54:09.133187 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:09.130717 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvv7c\" (UniqueName: \"kubernetes.io/projected/8e04ee6b-8e5b-4dc1-b001-184689f651e5-kube-api-access-bvv7c\") pod \"distinct-2-w4-063b46-kserve-7f485fc759-b65nm\" (UID: \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:54:09.166436 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:09.165996 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e04ee6b-8e5b-4dc1-b001-184689f651e5-tls-certs\") pod \"distinct-2-w4-063b46-kserve-7f485fc759-b65nm\" (UID: \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:54:09.184627 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:09.184538 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-dshm\") pod \"distinct-2-w4-063b46-kserve-7f485fc759-b65nm\" (UID: \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\") " pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:54:09.213125 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:09.212077 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:54:09.560453 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:09.560420 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm"] Oct 29 22:54:09.562958 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:54:09.562928 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e04ee6b_8e5b_4dc1_b001_184689f651e5.slice/crio-0859e3f69f22fc4c0e5ff74a4d36bedf45a6cf18e9789e89c70e82dbcf8c45f4 WatchSource:0}: Error finding container 0859e3f69f22fc4c0e5ff74a4d36bedf45a6cf18e9789e89c70e82dbcf8c45f4: Status 404 returned error can't find the container with id 0859e3f69f22fc4c0e5ff74a4d36bedf45a6cf18e9789e89c70e82dbcf8c45f4 Oct 29 22:54:09.594184 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:09.594153 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" event={"ID":"8e04ee6b-8e5b-4dc1-b001-184689f651e5","Type":"ContainerStarted","Data":"0859e3f69f22fc4c0e5ff74a4d36bedf45a6cf18e9789e89c70e82dbcf8c45f4"} Oct 29 22:54:10.601934 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:10.601892 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" event={"ID":"8e04ee6b-8e5b-4dc1-b001-184689f651e5","Type":"ContainerStarted","Data":"e230c235c18b884a72b4a6c9cc855193ba8086294b06f96112afedaa56a2ef34"} Oct 29 22:54:10.644595 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:10.644541 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podStartSLOduration=2.644526939 podStartE2EDuration="2.644526939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:10.641797323 +0000 UTC m=+704.582345233" watchObservedRunningTime="2026-10-29 22:54:10.644526939 +0000 UTC m=+704.585074848" Oct 29 22:54:11.213232 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:11.213166 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:54:14.784687 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.784653 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk"] Oct 29 22:54:14.788579 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.788556 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.819385 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.819357 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk"] Oct 29 22:54:14.857660 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.857622 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.857831 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.857748 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjxgh\" (UniqueName: \"kubernetes.io/projected/30e8d47c-27fb-40d8-86fb-eea11610630b-kube-api-access-zjxgh\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.857831 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.857809 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.857966 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.857904 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.857966 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.857931 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.858076 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.857976 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30e8d47c-27fb-40d8-86fb-eea11610630b-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.858076 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.858059 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.858179 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.858086 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.858179 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.858160 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.959193 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.959157 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjxgh\" (UniqueName: \"kubernetes.io/projected/30e8d47c-27fb-40d8-86fb-eea11610630b-kube-api-access-zjxgh\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.959367 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.959244 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.959367 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.959298 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.959367 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.959320 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.959367 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.959355 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30e8d47c-27fb-40d8-86fb-eea11610630b-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.959583 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.959373 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.959583 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.959398 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.959583 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.959453 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.959583 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.959486 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.959782 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.959731 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.959782 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.959731 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.959881 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.959829 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.959881 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.959849 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.960049 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.960026 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30e8d47c-27fb-40d8-86fb-eea11610630b-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.961829 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.961802 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.962097 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.962075 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.981841 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.981819 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjxgh\" (UniqueName: \"kubernetes.io/projected/30e8d47c-27fb-40d8-86fb-eea11610630b-kube-api-access-zjxgh\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:14.981935 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:14.981821 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:15.103087 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:15.103002 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:15.429788 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:15.429766 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk"] Oct 29 22:54:15.431304 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:54:15.431278 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e8d47c_27fb_40d8_86fb_eea11610630b.slice/crio-2a8fff725a18be122921d587581475f7fc5d7591b5306d92ae56effd00075110 WatchSource:0}: Error finding container 2a8fff725a18be122921d587581475f7fc5d7591b5306d92ae56effd00075110: Status 404 returned error can't find the container with id 2a8fff725a18be122921d587581475f7fc5d7591b5306d92ae56effd00075110 Oct 29 22:54:15.433893 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:15.433856 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:54:15.433995 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:15.433924 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:54:15.433995 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:15.433954 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:54:15.635782 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:15.635743 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" event={"ID":"30e8d47c-27fb-40d8-86fb-eea11610630b","Type":"ContainerStarted","Data":"511573a6f4f5a42b2a375ee02c1f383872f7cb22915a0563dd6ef36fef9b9b84"} Oct 29 22:54:15.635961 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:15.635809 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" event={"ID":"30e8d47c-27fb-40d8-86fb-eea11610630b","Type":"ContainerStarted","Data":"2a8fff725a18be122921d587581475f7fc5d7591b5306d92ae56effd00075110"} Oct 29 22:54:15.684919 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:15.684823 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" podStartSLOduration=1.684809427 podStartE2EDuration="1.684809427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:15.682539749 +0000 UTC m=+709.623087662" watchObservedRunningTime="2026-10-29 22:54:15.684809427 +0000 UTC m=+709.625357418" Oct 29 22:54:16.103630 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:16.103550 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:16.105271 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:16.105247 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:54:16.105344 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:16.105310 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" podUID="30e8d47c-27fb-40d8-86fb-eea11610630b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 29 22:54:17.104078 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:17.104045 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:54:17.104494 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:17.104093 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" podUID="30e8d47c-27fb-40d8-86fb-eea11610630b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 29 22:54:18.104031 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:18.103988 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:54:18.104231 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:18.104044 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" podUID="30e8d47c-27fb-40d8-86fb-eea11610630b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 29 22:54:19.103513 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:19.103478 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:54:19.103670 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:19.103538 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" podUID="30e8d47c-27fb-40d8-86fb-eea11610630b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 29 22:54:19.213614 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:19.213585 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:54:19.255119 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:19.255089 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:54:19.711513 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:19.711427 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:54:20.104107 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:20.104027 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:54:20.104289 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:20.104114 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" podUID="30e8d47c-27fb-40d8-86fb-eea11610630b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 29 22:54:21.104025 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:21.103987 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:54:21.104736 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:21.104699 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" podUID="30e8d47c-27fb-40d8-86fb-eea11610630b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 29 22:54:22.104179 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:22.104138 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:54:22.104786 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:22.104252 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" podUID="30e8d47c-27fb-40d8-86fb-eea11610630b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 29 22:54:23.108453 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:23.108422 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:23.108882 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:23.108815 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:23.109473 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:23.109453 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 22:54:23.884391 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:54:23.884354 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:54:31.980598 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:31.980559 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg"] Oct 29 22:54:38.628955 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:38.628918 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf"] Oct 29 22:54:38.634749 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:38.634720 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:54:38.660641 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:38.660616 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b2921b\"/\"distinct-2-w5-b2921b-kserve-self-signed-certs\"" Oct 29 22:54:38.663789 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:38.663768 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf"] Oct 29 22:54:38.683044 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:38.683020 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-model-cache\") pod \"distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf\" (UID: \"10c37b7b-4374-489e-8c71-653dd7229f60\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:54:38.683231 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:38.683050 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8n8s\" (UniqueName: \"kubernetes.io/projected/10c37b7b-4374-489e-8c71-653dd7229f60-kube-api-access-s8n8s\") pod \"distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf\" (UID: \"10c37b7b-4374-489e-8c71-653dd7229f60\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:54:38.683231 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:38.683080 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-dshm\") pod \"distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf\" (UID: \"10c37b7b-4374-489e-8c71-653dd7229f60\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:54:38.683231 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:38.683222 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c37b7b-4374-489e-8c71-653dd7229f60-tls-certs\") pod \"distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf\" (UID: \"10c37b7b-4374-489e-8c71-653dd7229f60\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:54:38.683407 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:38.683266 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-tmp-dir\") pod \"distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf\" (UID: \"10c37b7b-4374-489e-8c71-653dd7229f60\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:54:38.683407 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:38.683385 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-home\") pod \"distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf\" (UID: \"10c37b7b-4374-489e-8c71-653dd7229f60\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:54:38.784881 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:38.784846 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-model-cache\") pod \"distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf\" (UID: \"10c37b7b-4374-489e-8c71-653dd7229f60\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:54:38.785097 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:38.784888 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8n8s\" (UniqueName: \"kubernetes.io/projected/10c37b7b-4374-489e-8c71-653dd7229f60-kube-api-access-s8n8s\") pod \"distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf\" (UID: \"10c37b7b-4374-489e-8c71-653dd7229f60\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:54:38.785097 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:38.784925 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-dshm\") pod \"distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf\" (UID: \"10c37b7b-4374-489e-8c71-653dd7229f60\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:54:38.785097 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:38.784984 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c37b7b-4374-489e-8c71-653dd7229f60-tls-certs\") pod \"distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf\" (UID: \"10c37b7b-4374-489e-8c71-653dd7229f60\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:54:38.785097 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:38.785037 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-tmp-dir\") pod \"distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf\" (UID: \"10c37b7b-4374-489e-8c71-653dd7229f60\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:54:38.785097 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:38.785096 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-home\") pod \"distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf\" (UID: \"10c37b7b-4374-489e-8c71-653dd7229f60\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:54:38.785390 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:38.785357 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-model-cache\") pod \"distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf\" (UID: \"10c37b7b-4374-489e-8c71-653dd7229f60\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:54:38.785563 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:38.785532 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-tmp-dir\") pod \"distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf\" (UID: \"10c37b7b-4374-489e-8c71-653dd7229f60\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:54:38.785563 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:38.785557 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-home\") pod \"distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf\" (UID: \"10c37b7b-4374-489e-8c71-653dd7229f60\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:54:38.787474 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:38.787443 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-dshm\") pod \"distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf\" (UID: \"10c37b7b-4374-489e-8c71-653dd7229f60\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:54:38.787801 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:38.787780 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c37b7b-4374-489e-8c71-653dd7229f60-tls-certs\") pod \"distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf\" (UID: \"10c37b7b-4374-489e-8c71-653dd7229f60\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:54:38.811399 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:38.809413 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8n8s\" (UniqueName: \"kubernetes.io/projected/10c37b7b-4374-489e-8c71-653dd7229f60-kube-api-access-s8n8s\") pod \"distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf\" (UID: \"10c37b7b-4374-489e-8c71-653dd7229f60\") " pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:54:38.947931 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:38.947845 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:54:39.266443 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:54:39.266395 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10c37b7b_4374_489e_8c71_653dd7229f60.slice/crio-c6a721646d17f0edb24563d1bac3ce82f46fdc00c309b0b78db78aee44f749d4 WatchSource:0}: Error finding container c6a721646d17f0edb24563d1bac3ce82f46fdc00c309b0b78db78aee44f749d4: Status 404 returned error can't find the container with id c6a721646d17f0edb24563d1bac3ce82f46fdc00c309b0b78db78aee44f749d4 Oct 29 22:54:39.267633 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:39.267606 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf"] Oct 29 22:54:39.844471 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:39.844431 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" event={"ID":"10c37b7b-4374-489e-8c71-653dd7229f60","Type":"ContainerStarted","Data":"087d484e5ba46a82ce19d13ec113c778fca35016cb7257d93e3faccc48e99a3a"} Oct 29 22:54:39.844884 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:39.844478 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" event={"ID":"10c37b7b-4374-489e-8c71-653dd7229f60","Type":"ContainerStarted","Data":"c6a721646d17f0edb24563d1bac3ce82f46fdc00c309b0b78db78aee44f749d4"} Oct 29 22:54:39.888329 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:39.888276 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podStartSLOduration=1.88826104 podStartE2EDuration="1.88826104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:39.886056128 +0000 UTC m=+733.826604038" watchObservedRunningTime="2026-10-29 22:54:39.88826104 +0000 UTC m=+733.828808985" Oct 29 22:54:39.948170 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:39.948142 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:54:47.013474 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.013434 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" podUID="18e49082-9069-46f6-ba3c-409196298ce4" containerName="main" containerID="cri-o://32d1a4322eb69b25bb7bc85e1759151c1b9af40232eaeb06d6de50675d514bc1" gracePeriod=45 Oct 29 22:54:47.108608 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.108561 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" podUID="18e49082-9069-46f6-ba3c-409196298ce4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": dial tcp 10.133.0.62:8000: connect: connection refused" Oct 29 22:54:47.282888 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.282822 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:54:47.387250 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.387220 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-model-cache\") pod \"18e49082-9069-46f6-ba3c-409196298ce4\" (UID: \"18e49082-9069-46f6-ba3c-409196298ce4\") " Oct 29 22:54:47.387444 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.387260 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-home\" (UniqueName: \"kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-home\") pod \"18e49082-9069-46f6-ba3c-409196298ce4\" (UID: \"18e49082-9069-46f6-ba3c-409196298ce4\") " Oct 29 22:54:47.387444 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.387280 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18e49082-9069-46f6-ba3c-409196298ce4-tls-certs\" (UniqueName: \"kubernetes.io/secret/18e49082-9069-46f6-ba3c-409196298ce4-tls-certs\") pod \"18e49082-9069-46f6-ba3c-409196298ce4\" (UID: \"18e49082-9069-46f6-ba3c-409196298ce4\") " Oct 29 22:54:47.387444 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.387357 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-tmp-dir\") pod \"18e49082-9069-46f6-ba3c-409196298ce4\" (UID: \"18e49082-9069-46f6-ba3c-409196298ce4\") " Oct 29 22:54:47.387444 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.387403 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-dshm\") pod \"18e49082-9069-46f6-ba3c-409196298ce4\" (UID: \"18e49082-9069-46f6-ba3c-409196298ce4\") " Oct 29 22:54:47.387444 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.387444 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e49082-9069-46f6-ba3c-409196298ce4-kube-api-access-44c9x\" (UniqueName: \"kubernetes.io/projected/18e49082-9069-46f6-ba3c-409196298ce4-kube-api-access-44c9x\") pod \"18e49082-9069-46f6-ba3c-409196298ce4\" (UID: \"18e49082-9069-46f6-ba3c-409196298ce4\") " Oct 29 22:54:47.387727 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.387537 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-home" pod "18e49082-9069-46f6-ba3c-409196298ce4" (UID: "18e49082-9069-46f6-ba3c-409196298ce4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:54:47.387727 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.387549 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-model-cache" pod "18e49082-9069-46f6-ba3c-409196298ce4" (UID: "18e49082-9069-46f6-ba3c-409196298ce4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:54:47.387727 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.387658 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-tmp-dir" pod "18e49082-9069-46f6-ba3c-409196298ce4" (UID: "18e49082-9069-46f6-ba3c-409196298ce4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:54:47.387944 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.387917 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:54:47.387944 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.387945 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:54:47.388921 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.387958 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:54:47.389802 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.389770 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-dshm" pod "18e49082-9069-46f6-ba3c-409196298ce4" (UID: "18e49082-9069-46f6-ba3c-409196298ce4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:54:47.389900 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.389855 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18e49082-9069-46f6-ba3c-409196298ce4-tls-certs" pod "18e49082-9069-46f6-ba3c-409196298ce4" (UID: "18e49082-9069-46f6-ba3c-409196298ce4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:54:47.389900 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.389871 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e49082-9069-46f6-ba3c-409196298ce4-kube-api-access-44c9x" pod "18e49082-9069-46f6-ba3c-409196298ce4" (UID: "18e49082-9069-46f6-ba3c-409196298ce4"). InnerVolumeSpecName "kube-api-access-44c9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:54:47.489429 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.489394 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e49082-9069-46f6-ba3c-409196298ce4-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:54:47.489429 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.489424 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44c9x\" (UniqueName: \"kubernetes.io/projected/18e49082-9069-46f6-ba3c-409196298ce4-kube-api-access-44c9x\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:54:47.489429 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.489435 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e49082-9069-46f6-ba3c-409196298ce4-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:54:47.895872 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.895833 4821 generic.go:356] "Generic (PLEG): container finished" podID="18e49082-9069-46f6-ba3c-409196298ce4" containerID="32d1a4322eb69b25bb7bc85e1759151c1b9af40232eaeb06d6de50675d514bc1" exitCode=0 Oct 29 22:54:47.896070 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.895881 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" Oct 29 22:54:47.896070 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.895917 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" event={"ID":"18e49082-9069-46f6-ba3c-409196298ce4","Type":"ContainerDied","Data":"32d1a4322eb69b25bb7bc85e1759151c1b9af40232eaeb06d6de50675d514bc1"} Oct 29 22:54:47.896070 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.895966 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg" event={"ID":"18e49082-9069-46f6-ba3c-409196298ce4","Type":"ContainerDied","Data":"6369901ad70feb9ef1158f967eadbc8feb01b50cd215973bdcebdcb58d4267c2"} Oct 29 22:54:47.896070 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.895984 4821 scope.go:122] "RemoveContainer" containerID="32d1a4322eb69b25bb7bc85e1759151c1b9af40232eaeb06d6de50675d514bc1" Oct 29 22:54:47.906126 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.906110 4821 scope.go:122] "RemoveContainer" containerID="32d1a4322eb69b25bb7bc85e1759151c1b9af40232eaeb06d6de50675d514bc1" Oct 29 22:54:47.906394 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:54:47.906378 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32d1a4322eb69b25bb7bc85e1759151c1b9af40232eaeb06d6de50675d514bc1\": container with ID starting with 32d1a4322eb69b25bb7bc85e1759151c1b9af40232eaeb06d6de50675d514bc1 not found: ID does not exist" containerID="32d1a4322eb69b25bb7bc85e1759151c1b9af40232eaeb06d6de50675d514bc1" Oct 29 22:54:47.906445 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.906403 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32d1a4322eb69b25bb7bc85e1759151c1b9af40232eaeb06d6de50675d514bc1"} err="failed to get container status \"32d1a4322eb69b25bb7bc85e1759151c1b9af40232eaeb06d6de50675d514bc1\": rpc error: code = NotFound desc = could not find container \"32d1a4322eb69b25bb7bc85e1759151c1b9af40232eaeb06d6de50675d514bc1\": container with ID starting with 32d1a4322eb69b25bb7bc85e1759151c1b9af40232eaeb06d6de50675d514bc1 not found: ID does not exist" Oct 29 22:54:47.946361 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.946328 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg"] Oct 29 22:54:47.958883 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:47.958848 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/e2e-conflict-a-2f81a573-kserve-6fbff466bb-qvqxg"] Oct 29 22:54:48.169990 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:48.169896 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e49082-9069-46f6-ba3c-409196298ce4" path="/var/lib/kubelet/pods/18e49082-9069-46f6-ba3c-409196298ce4/volumes" Oct 29 22:54:48.948189 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:48.948156 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:54:48.961998 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:48.961976 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:54:49.920337 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:49.920309 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:54:51.404925 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.404890 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb"] Oct 29 22:54:51.405386 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.405372 4821 cpu_manager.go:414] "containerMap: removing container" podUID="18e49082-9069-46f6-ba3c-409196298ce4" containerName="main" Oct 29 22:54:51.405434 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.405388 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e49082-9069-46f6-ba3c-409196298ce4" containerName="main" Oct 29 22:54:51.405495 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.405482 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e49082-9069-46f6-ba3c-409196298ce4" containerName="main" Oct 29 22:54:51.410263 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.410237 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.451875 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.451843 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb"] Oct 29 22:54:51.452120 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.452108 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc"] Oct 29 22:54:51.456453 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.456419 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.474737 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.474707 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc"] Oct 29 22:54:51.533821 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.533785 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.533984 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.533827 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.533984 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.533850 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.533984 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.533959 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.534138 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.533989 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.534138 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.534010 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.534288 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.534152 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.534288 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.534192 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.534288 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.534269 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.534481 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.534329 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crpl7\" (UniqueName: \"kubernetes.io/projected/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-kube-api-access-crpl7\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.534481 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.534369 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.534481 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.534406 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.534481 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.534457 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.534613 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.534484 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.534613 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.534550 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb9wk\" (UniqueName: \"kubernetes.io/projected/ea79c88e-57ac-4f8f-a392-bf31d397cae8-kube-api-access-tb9wk\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.534613 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.534595 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.534712 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.534643 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.534748 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.534707 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.635764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.635729 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.635936 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.635772 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.635936 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.635795 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crpl7\" (UniqueName: \"kubernetes.io/projected/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-kube-api-access-crpl7\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.635936 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.635828 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.635936 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.635852 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.635936 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.635882 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.635936 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.635915 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.636287 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.635977 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb9wk\" (UniqueName: \"kubernetes.io/projected/ea79c88e-57ac-4f8f-a392-bf31d397cae8-kube-api-access-tb9wk\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.636287 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.636005 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.636287 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.636060 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.636287 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.636099 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.636287 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.636126 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.636287 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.636155 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.636287 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.636181 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.636287 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.636241 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.636287 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.636251 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.636287 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.636245 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.636287 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.636286 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.636856 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.636314 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.636856 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.636386 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.636856 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.636530 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.636856 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.636573 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.636856 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.636682 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.637122 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.636910 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.637122 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.636969 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.637246 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.637135 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.637246 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.637150 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.637406 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.637379 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.638560 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.638541 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.638998 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.638979 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.639347 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.639328 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.639430 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.639364 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.658969 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.658897 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crpl7\" (UniqueName: \"kubernetes.io/projected/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-kube-api-access-crpl7\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.658969 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.658956 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.659231 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.659182 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:51.659282 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.659272 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb9wk\" (UniqueName: \"kubernetes.io/projected/ea79c88e-57ac-4f8f-a392-bf31d397cae8-kube-api-access-tb9wk\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.722844 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.722811 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:51.776574 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:51.775797 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:52.044398 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:52.044369 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb"] Oct 29 22:54:52.044932 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:54:52.044908 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea79c88e_57ac_4f8f_a392_bf31d397cae8.slice/crio-20faf028eecca7cfdfc0c41bbda5d9b578b7e72e098ab78c897f0036a910ffbf WatchSource:0}: Error finding container 20faf028eecca7cfdfc0c41bbda5d9b578b7e72e098ab78c897f0036a910ffbf: Status 404 returned error can't find the container with id 20faf028eecca7cfdfc0c41bbda5d9b578b7e72e098ab78c897f0036a910ffbf Oct 29 22:54:52.047475 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:52.047443 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:54:52.047574 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:52.047516 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:54:52.047574 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:52.047545 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:54:52.099689 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:52.099668 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc"] Oct 29 22:54:52.101484 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:54:52.101459 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8d67b9_8725_4c23_a125_9bfdf3ff4917.slice/crio-056aeb5289ac31cee7c3aa6b1b2ea48330a78a7f3aba1bced9d538ad93b9ad51 WatchSource:0}: Error finding container 056aeb5289ac31cee7c3aa6b1b2ea48330a78a7f3aba1bced9d538ad93b9ad51: Status 404 returned error can't find the container with id 056aeb5289ac31cee7c3aa6b1b2ea48330a78a7f3aba1bced9d538ad93b9ad51 Oct 29 22:54:52.104160 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:52.104121 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:54:52.104340 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:52.104236 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:54:52.104340 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:52.104285 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:54:52.930669 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:52.930625 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" event={"ID":"fd8d67b9-8725-4c23-a125-9bfdf3ff4917","Type":"ContainerStarted","Data":"313a4ab372e3140a6d4933d354e7c6a35cfc98a8c03942011d79430fbaf6940a"} Oct 29 22:54:52.931368 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:52.930698 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" event={"ID":"fd8d67b9-8725-4c23-a125-9bfdf3ff4917","Type":"ContainerStarted","Data":"056aeb5289ac31cee7c3aa6b1b2ea48330a78a7f3aba1bced9d538ad93b9ad51"} Oct 29 22:54:52.932728 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:52.932700 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" event={"ID":"ea79c88e-57ac-4f8f-a392-bf31d397cae8","Type":"ContainerStarted","Data":"bd48930866127a1b2060e2d12207b6ce7e34c63f1e1f1ff9f9cbce3544fb1239"} Oct 29 22:54:52.934904 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:52.934880 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" event={"ID":"ea79c88e-57ac-4f8f-a392-bf31d397cae8","Type":"ContainerStarted","Data":"20faf028eecca7cfdfc0c41bbda5d9b578b7e72e098ab78c897f0036a910ffbf"} Oct 29 22:54:53.007978 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:53.007924 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" podStartSLOduration=2.007909445 podStartE2EDuration="2.007909445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:52.994774688 +0000 UTC m=+746.935322598" watchObservedRunningTime="2026-10-29 22:54:53.007909445 +0000 UTC m=+746.948457355" Oct 29 22:54:53.044224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:53.043158 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" podStartSLOduration=2.04313816 podStartE2EDuration="2.04313816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:53.038698324 +0000 UTC m=+746.979246270" watchObservedRunningTime="2026-10-29 22:54:53.04313816 +0000 UTC m=+746.983686071" Oct 29 22:54:53.725647 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:53.725607 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:54:53.730357 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:53.730321 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 29 22:54:53.730608 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:53.730581 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" podUID="ea79c88e-57ac-4f8f-a392-bf31d397cae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 29 22:54:53.776737 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:53.776702 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:54:53.778157 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:53.778132 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 29 22:54:53.778290 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:53.778188 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" podUID="fd8d67b9-8725-4c23-a125-9bfdf3ff4917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 29 22:54:53.893337 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:54:53.893299 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:54:54.725227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:54.724586 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 29 22:54:54.725227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:54.724648 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" podUID="ea79c88e-57ac-4f8f-a392-bf31d397cae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 29 22:54:54.776616 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:54.776586 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 29 22:54:54.776837 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:54.776818 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" podUID="fd8d67b9-8725-4c23-a125-9bfdf3ff4917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 29 22:54:55.729224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:55.726572 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 29 22:54:55.729224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:55.726638 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" podUID="ea79c88e-57ac-4f8f-a392-bf31d397cae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 29 22:54:55.777224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:55.776680 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 29 22:54:55.777224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:55.776757 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" podUID="fd8d67b9-8725-4c23-a125-9bfdf3ff4917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 29 22:54:56.360683 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.360634 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b"] Oct 29 22:54:56.366998 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.366972 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:54:56.388927 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.388891 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b"] Oct 29 22:54:56.393904 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.393562 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-31226c\"/\"distinct-2-w6-31226c-kserve-self-signed-certs\"" Oct 29 22:54:56.513333 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.512862 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n84vm\" (UniqueName: \"kubernetes.io/projected/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-kube-api-access-n84vm\") pod \"distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b\" (UID: \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:54:56.513333 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.512912 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-home\") pod \"distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b\" (UID: \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:54:56.513333 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.513040 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-tmp-dir\") pod \"distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b\" (UID: \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:54:56.513333 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.513108 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-model-cache\") pod \"distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b\" (UID: \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:54:56.513333 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.513130 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-tls-certs\") pod \"distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b\" (UID: \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:54:56.513333 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.513173 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-dshm\") pod \"distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b\" (UID: \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:54:56.614300 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.614170 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-tmp-dir\") pod \"distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b\" (UID: \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:54:56.614474 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.614305 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-model-cache\") pod \"distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b\" (UID: \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:54:56.614474 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.614359 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-tls-certs\") pod \"distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b\" (UID: \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:54:56.614474 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.614422 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-dshm\") pod \"distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b\" (UID: \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:54:56.614731 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.614505 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n84vm\" (UniqueName: \"kubernetes.io/projected/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-kube-api-access-n84vm\") pod \"distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b\" (UID: \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:54:56.614731 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.614532 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-home\") pod \"distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b\" (UID: \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:54:56.619250 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.615043 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-home\") pod \"distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b\" (UID: \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:54:56.619250 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.615331 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-tmp-dir\") pod \"distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b\" (UID: \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:54:56.619250 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.615568 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-model-cache\") pod \"distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b\" (UID: \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:54:56.656226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.656013 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-tls-certs\") pod \"distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b\" (UID: \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:54:56.662525 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.658476 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-dshm\") pod \"distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b\" (UID: \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:54:56.662525 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.661555 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n84vm\" (UniqueName: \"kubernetes.io/projected/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-kube-api-access-n84vm\") pod \"distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b\" (UID: \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\") " pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:54:56.696229 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.692422 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:54:56.724225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.723584 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 29 22:54:56.724225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.723644 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" podUID="ea79c88e-57ac-4f8f-a392-bf31d397cae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 29 22:54:56.779225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.778643 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 29 22:54:56.779225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:56.778745 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" podUID="fd8d67b9-8725-4c23-a125-9bfdf3ff4917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 29 22:54:57.183991 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:57.183968 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b"] Oct 29 22:54:57.186931 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:54:57.186900 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf00ff5fc_65d2_4873_9bd5_60feb93b93d8.slice/crio-c359dbbc89fb4feba83093e8e80f003fc12c272b840b27b7817b5081c740d3aa WatchSource:0}: Error finding container c359dbbc89fb4feba83093e8e80f003fc12c272b840b27b7817b5081c740d3aa: Status 404 returned error can't find the container with id c359dbbc89fb4feba83093e8e80f003fc12c272b840b27b7817b5081c740d3aa Oct 29 22:54:57.734228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:57.731666 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 29 22:54:57.734228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:57.731729 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" podUID="ea79c88e-57ac-4f8f-a392-bf31d397cae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 29 22:54:57.776528 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:57.776492 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 29 22:54:57.776700 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:57.776542 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" podUID="fd8d67b9-8725-4c23-a125-9bfdf3ff4917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 29 22:54:58.004831 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:58.004752 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" event={"ID":"f00ff5fc-65d2-4873-9bd5-60feb93b93d8","Type":"ContainerStarted","Data":"0f8c172db814ec09c3af4f4e6b891049a79cac4fcb51690ab2d50a63547b4264"} Oct 29 22:54:58.004831 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:58.004793 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" event={"ID":"f00ff5fc-65d2-4873-9bd5-60feb93b93d8","Type":"ContainerStarted","Data":"c359dbbc89fb4feba83093e8e80f003fc12c272b840b27b7817b5081c740d3aa"} Oct 29 22:54:58.693213 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:58.693176 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:54:58.730649 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:58.730611 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 29 22:54:58.730975 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:58.730937 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" podUID="ea79c88e-57ac-4f8f-a392-bf31d397cae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 29 22:54:58.777376 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:58.777337 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 29 22:54:58.777522 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:54:58.777424 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" podUID="fd8d67b9-8725-4c23-a125-9bfdf3ff4917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 29 22:55:00.066776 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:00.066741 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:55:00.067484 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:00.067462 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:55:00.069835 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:00.069809 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 22:55:00.127250 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:00.127179 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podStartSLOduration=4.12716604 podStartE2EDuration="4.12716604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:58.06557416 +0000 UTC m=+752.006122072" watchObservedRunningTime="2026-10-29 22:55:00.12716604 +0000 UTC m=+754.067713950" Oct 29 22:55:00.269365 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:00.269326 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:55:00.270748 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:00.270381 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:55:00.271279 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:00.271255 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 22:55:01.807243 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:01.806296 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:01.919224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:01.918293 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:02.007226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:02.006284 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:02.126235 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:02.123583 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:02.222865 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:02.222823 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:02.268228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:02.267309 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:02.399394 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:02.399291 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:55:02.462223 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:02.461364 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 22:55:02.462223 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:02.462144 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:06.693600 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:06.693568 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:55:06.711655 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:06.711631 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:55:07.148993 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:07.148967 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:55:11.900373 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:11.900337 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg"] Oct 29 22:55:11.900884 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:11.900574 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" podUID="5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c" containerName="istio-proxy" containerID="cri-o://9cd5b0dd2211876c57e9396d401d7cf2b6664e42a779ae57fb885c397ab046e4" gracePeriod=30 Oct 29 22:55:17.173054 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.172987 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:55:17.200370 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.200341 4821 generic.go:356] "Generic (PLEG): container finished" podID="5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c" containerID="9cd5b0dd2211876c57e9396d401d7cf2b6664e42a779ae57fb885c397ab046e4" exitCode=0 Oct 29 22:55:17.200527 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.200391 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" Oct 29 22:55:17.200527 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.200428 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" event={"ID":"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c","Type":"ContainerDied","Data":"9cd5b0dd2211876c57e9396d401d7cf2b6664e42a779ae57fb885c397ab046e4"} Oct 29 22:55:17.200527 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.200486 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg" event={"ID":"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c","Type":"ContainerDied","Data":"99bd5a8ebf8a02fc78f07b4a310367c2c09427fd4cb9cfe74c160af9527f845a"} Oct 29 22:55:17.200527 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.200513 4821 scope.go:122] "RemoveContainer" containerID="9cd5b0dd2211876c57e9396d401d7cf2b6664e42a779ae57fb885c397ab046e4" Oct 29 22:55:17.210518 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.210495 4821 scope.go:122] "RemoveContainer" containerID="9cd5b0dd2211876c57e9396d401d7cf2b6664e42a779ae57fb885c397ab046e4" Oct 29 22:55:17.210864 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:55:17.210825 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cd5b0dd2211876c57e9396d401d7cf2b6664e42a779ae57fb885c397ab046e4\": container with ID starting with 9cd5b0dd2211876c57e9396d401d7cf2b6664e42a779ae57fb885c397ab046e4 not found: ID does not exist" containerID="9cd5b0dd2211876c57e9396d401d7cf2b6664e42a779ae57fb885c397ab046e4" Oct 29 22:55:17.210983 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.210867 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cd5b0dd2211876c57e9396d401d7cf2b6664e42a779ae57fb885c397ab046e4"} err="failed to get container status \"9cd5b0dd2211876c57e9396d401d7cf2b6664e42a779ae57fb885c397ab046e4\": rpc error: code = NotFound desc = could not find container \"9cd5b0dd2211876c57e9396d401d7cf2b6664e42a779ae57fb885c397ab046e4\": container with ID starting with 9cd5b0dd2211876c57e9396d401d7cf2b6664e42a779ae57fb885c397ab046e4 not found: ID does not exist" Oct 29 22:55:17.291168 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.291136 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-data\") pod \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " Oct 29 22:55:17.291342 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.291274 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istiod-ca-cert\") pod \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " Oct 29 22:55:17.291342 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.291299 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-token\" (UniqueName: \"kubernetes.io/projected/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-token\") pod \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " Oct 29 22:55:17.291342 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.291320 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-kube-api-access-g5tm8\" (UniqueName: \"kubernetes.io/projected/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-kube-api-access-g5tm8\") pod \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " Oct 29 22:55:17.291525 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.291349 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-podinfo\") pod \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " Oct 29 22:55:17.291525 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.291393 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-envoy\") pod \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " Oct 29 22:55:17.291525 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.291437 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-workload-certs\") pod \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " Oct 29 22:55:17.291525 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.291465 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-data" pod "5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c" (UID: "5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:55:17.291525 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.291510 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-credential-socket\") pod \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " Oct 29 22:55:17.291767 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.291538 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-workload-socket\") pod \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\" (UID: \"5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c\") " Oct 29 22:55:17.291822 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.291765 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-workload-certs" pod "5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c" (UID: "5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:55:17.291822 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.291786 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istiod-ca-cert" pod "5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c" (UID: "5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:55:17.291990 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.291938 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-credential-socket" pod "5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c" (UID: "5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:55:17.292126 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.292005 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-workload-socket" pod "5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c" (UID: "5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:55:17.292126 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.292069 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:55:17.292126 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.292090 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:55:17.292126 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.292104 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:55:17.292126 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.292115 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:55:17.293815 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.293789 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-token" pod "5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c" (UID: "5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:55:17.294114 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.294099 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-podinfo" pod "5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c" (UID: "5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:55:17.294269 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.294254 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-kube-api-access-g5tm8" pod "5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c" (UID: "5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c"). InnerVolumeSpecName "kube-api-access-g5tm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:55:17.294398 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.294376 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-envoy" pod "5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c" (UID: "5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:55:17.393518 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.393399 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:55:17.393518 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.393439 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:55:17.393518 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.393455 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:55:17.393518 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.393468 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:55:17.393518 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.393483 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5tm8\" (UniqueName: \"kubernetes.io/projected/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c-kube-api-access-g5tm8\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:55:17.547967 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.547938 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg"] Oct 29 22:55:17.558848 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:17.558821 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-k4pqg"] Oct 29 22:55:18.169851 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:18.169813 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c" path="/var/lib/kubelet/pods/5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c/volumes" Oct 29 22:55:23.017356 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.017319 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg"] Oct 29 22:55:23.017801 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.017786 4821 cpu_manager.go:414] "containerMap: removing container" podUID="5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c" containerName="istio-proxy" Oct 29 22:55:23.017844 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.017802 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c" containerName="istio-proxy" Oct 29 22:55:23.017912 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.017903 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="5402bc5f-c452-45b0-bfe0-ba0a3fc5d83c" containerName="istio-proxy" Oct 29 22:55:23.023179 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.023143 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.037564 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.037543 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-b2921b-openshift-default-dockercfg-652vx\"" Oct 29 22:55:23.054301 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.054265 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg"] Oct 29 22:55:23.163145 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.163112 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.163324 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.163158 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.163324 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.163282 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d7791f-31f4-4552-91d0-120f640ad1fa-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.163324 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.163313 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n9m5\" (UniqueName: \"kubernetes.io/projected/68d7791f-31f4-4552-91d0-120f640ad1fa-kube-api-access-2n9m5\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.163444 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.163332 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.163444 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.163417 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.163512 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.163466 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.163512 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.163486 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.163636 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.163622 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.264884 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.264846 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.265092 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.264905 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.265092 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.264989 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.265284 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.265136 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.265396 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.265373 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.265464 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.265390 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.265515 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.265467 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.265571 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.265515 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.265635 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.265377 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.265635 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.265598 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d7791f-31f4-4552-91d0-120f640ad1fa-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.265743 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.265646 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n9m5\" (UniqueName: \"kubernetes.io/projected/68d7791f-31f4-4552-91d0-120f640ad1fa-kube-api-access-2n9m5\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.266400 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.265837 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.266400 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.265894 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.266400 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.266366 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d7791f-31f4-4552-91d0-120f640ad1fa-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.267843 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.267788 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.268095 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.268077 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.287628 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.287596 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.287799 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.287780 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n9m5\" (UniqueName: \"kubernetes.io/projected/68d7791f-31f4-4552-91d0-120f640ad1fa-kube-api-access-2n9m5\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.336382 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.336350 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:23.657569 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.657543 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg"] Oct 29 22:55:23.658703 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:55:23.658669 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d7791f_31f4_4552_91d0_120f640ad1fa.slice/crio-167aa22c3fc2510e010c60a7b1737f2a4f47ec3e857f3f5d121bf9532827a2fc WatchSource:0}: Error finding container 167aa22c3fc2510e010c60a7b1737f2a4f47ec3e857f3f5d121bf9532827a2fc: Status 404 returned error can't find the container with id 167aa22c3fc2510e010c60a7b1737f2a4f47ec3e857f3f5d121bf9532827a2fc Oct 29 22:55:23.661043 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.661013 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:55:23.661116 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.661079 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:55:23.661116 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:23.661108 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:55:23.887554 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:55:23.887518 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:55:24.248059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:24.248024 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" event={"ID":"68d7791f-31f4-4552-91d0-120f640ad1fa","Type":"ContainerStarted","Data":"3d63bf0b80f83ed78e7cc83b12a8ca66f5b207ad0b5b44098d3d9531da1ed3bd"} Oct 29 22:55:24.248576 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:24.248079 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" event={"ID":"68d7791f-31f4-4552-91d0-120f640ad1fa","Type":"ContainerStarted","Data":"167aa22c3fc2510e010c60a7b1737f2a4f47ec3e857f3f5d121bf9532827a2fc"} Oct 29 22:55:24.294392 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:24.294346 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" podStartSLOduration=2.29433182 podStartE2EDuration="2.29433182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:24.291058757 +0000 UTC m=+778.231606666" watchObservedRunningTime="2026-10-29 22:55:24.29433182 +0000 UTC m=+778.234879731" Oct 29 22:55:24.336781 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:24.336749 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:24.338494 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:24.338468 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 29 22:55:24.338616 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:24.338528 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" podUID="68d7791f-31f4-4552-91d0-120f640ad1fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 29 22:55:25.338225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:25.337658 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 29 22:55:25.338225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:25.337723 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" podUID="68d7791f-31f4-4552-91d0-120f640ad1fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 29 22:55:26.346576 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:26.346541 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 29 22:55:26.347035 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:26.346601 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" podUID="68d7791f-31f4-4552-91d0-120f640ad1fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 29 22:55:27.340675 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:27.340634 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 29 22:55:27.340963 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:27.340931 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" podUID="68d7791f-31f4-4552-91d0-120f640ad1fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 29 22:55:28.059228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:28.054291 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:28.128225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:28.127455 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:28.143230 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:28.139245 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 29 22:55:28.221329 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:28.221282 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:28.267393 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:28.267346 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:28.363226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:28.361754 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 29 22:55:28.363226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:28.361817 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" podUID="68d7791f-31f4-4552-91d0-120f640ad1fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 29 22:55:28.393643 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:28.393587 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:28.448548 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:28.447317 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:28.463816 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:28.463688 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:28.626454 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:28.626342 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:28.710140 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:28.709303 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:28.837336 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:28.837287 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 29 22:55:29.010426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:29.008292 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:29.062229 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:29.061326 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:29.062758 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:29.062327 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:55:29.166227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:29.165647 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:29.166227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:29.166041 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 29 22:55:29.172262 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:29.166660 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:55:29.172262 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:29.166741 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:55:29.233880 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:29.233840 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:29.316910 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:29.291312 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:29.317193 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:29.291373 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 22:55:29.340069 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:29.340037 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 29 22:55:29.340338 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:29.340309 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" podUID="68d7791f-31f4-4552-91d0-120f640ad1fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 29 22:55:29.340586 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:29.340569 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:55:29.340772 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:29.340759 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:55:29.401462 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:29.399307 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:29.401462 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:29.399843 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:55:29.474644 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:29.474604 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:29.475332 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:29.475310 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:55:29.713117 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:29.713079 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 29 22:55:29.720365 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:29.720331 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:55:29.868562 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:29.865843 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:29.873727 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:29.873706 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:55:29.907224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:29.903552 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:55:29.934672 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:29.934628 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 29 22:55:30.001115 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:30.001028 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:55:30.099401 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:30.099361 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 29 22:55:30.107935 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:30.107912 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:55:30.175650 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:30.174627 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:55:30.207106 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:30.207079 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:55:30.226235 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:30.215328 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:30.226235 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:30.215794 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:30.260136 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:30.259887 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:55:30.362381 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:30.360342 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 29 22:55:30.362381 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:30.360442 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" podUID="68d7791f-31f4-4552-91d0-120f640ad1fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 29 22:55:30.364128 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:30.364095 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:30.378545 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:30.378499 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:30.615287 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:30.610324 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:55:30.615287 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:30.613626 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:55:30.706700 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:30.706366 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:55:31.337632 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:31.337596 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 29 22:55:31.338267 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:31.338232 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" podUID="68d7791f-31f4-4552-91d0-120f640ad1fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 29 22:55:32.340820 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:32.340719 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 29 22:55:32.340820 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:32.340805 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" podUID="68d7791f-31f4-4552-91d0-120f640ad1fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 29 22:55:34.337056 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:34.337014 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:55:34.337668 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:34.337096 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" podUID="68d7791f-31f4-4552-91d0-120f640ad1fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:34.385816 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:34.385788 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:34.386182 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:34.386163 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:34.386874 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:34.386857 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 22:55:35.000689 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:35.000654 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f42f78-cfknt"] Oct 29 22:55:35.010483 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:35.010454 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f42f78-cfknt" Oct 29 22:55:35.024402 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:35.024374 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f42f78-cfknt"] Oct 29 22:55:35.031955 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:35.031932 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:55:35.031955 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:35.031939 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 22:55:35.109634 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:35.109599 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58366efd-cd68-466c-bc65-154613ed0ffe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0f42f78-cfknt\" (UID: \"58366efd-cd68-466c-bc65-154613ed0ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f42f78-cfknt" Oct 29 22:55:35.109860 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:35.109835 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58366efd-cd68-466c-bc65-154613ed0ffe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0f42f78-cfknt\" (UID: \"58366efd-cd68-466c-bc65-154613ed0ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f42f78-cfknt" Oct 29 22:55:35.109913 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:35.109876 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdnvd\" (UniqueName: \"kubernetes.io/projected/58366efd-cd68-466c-bc65-154613ed0ffe-kube-api-access-zdnvd\") pod \"maas-api-revoke-sub-f0f42f78-cfknt\" (UID: \"58366efd-cd68-466c-bc65-154613ed0ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f42f78-cfknt" Oct 29 22:55:35.210861 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:35.210818 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58366efd-cd68-466c-bc65-154613ed0ffe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0f42f78-cfknt\" (UID: \"58366efd-cd68-466c-bc65-154613ed0ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f42f78-cfknt" Oct 29 22:55:35.210861 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:35.210860 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdnvd\" (UniqueName: \"kubernetes.io/projected/58366efd-cd68-466c-bc65-154613ed0ffe-kube-api-access-zdnvd\") pod \"maas-api-revoke-sub-f0f42f78-cfknt\" (UID: \"58366efd-cd68-466c-bc65-154613ed0ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f42f78-cfknt" Oct 29 22:55:35.211119 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:35.210950 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58366efd-cd68-466c-bc65-154613ed0ffe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0f42f78-cfknt\" (UID: \"58366efd-cd68-466c-bc65-154613ed0ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f42f78-cfknt" Oct 29 22:55:35.211930 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:35.211907 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58366efd-cd68-466c-bc65-154613ed0ffe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0f42f78-cfknt\" (UID: \"58366efd-cd68-466c-bc65-154613ed0ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f42f78-cfknt" Oct 29 22:55:35.217640 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:35.217610 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58366efd-cd68-466c-bc65-154613ed0ffe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0f42f78-cfknt\" (UID: \"58366efd-cd68-466c-bc65-154613ed0ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f42f78-cfknt" Oct 29 22:55:35.234131 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:35.234104 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdnvd\" (UniqueName: \"kubernetes.io/projected/58366efd-cd68-466c-bc65-154613ed0ffe-kube-api-access-zdnvd\") pod \"maas-api-revoke-sub-f0f42f78-cfknt\" (UID: \"58366efd-cd68-466c-bc65-154613ed0ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f42f78-cfknt" Oct 29 22:55:35.324847 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:35.324764 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f42f78-cfknt" Oct 29 22:55:35.647821 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:35.647789 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f42f78-cfknt"] Oct 29 22:55:35.649008 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:55:35.648980 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58366efd_cd68_466c_bc65_154613ed0ffe.slice/crio-43280f9dd02178f06f0886cc39a2880b9a4dccc6cbc1a4c2db5f50f29e30ad34 WatchSource:0}: Error finding container 43280f9dd02178f06f0886cc39a2880b9a4dccc6cbc1a4c2db5f50f29e30ad34: Status 404 returned error can't find the container with id 43280f9dd02178f06f0886cc39a2880b9a4dccc6cbc1a4c2db5f50f29e30ad34 Oct 29 22:55:35.746874 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:35.746839 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f42f78-cfknt" event={"ID":"58366efd-cd68-466c-bc65-154613ed0ffe","Type":"ContainerStarted","Data":"43280f9dd02178f06f0886cc39a2880b9a4dccc6cbc1a4c2db5f50f29e30ad34"} Oct 29 22:55:36.755531 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:36.755490 4821 generic.go:356] "Generic (PLEG): container finished" podID="58366efd-cd68-466c-bc65-154613ed0ffe" containerID="b537fac20fec3d960c92a6feea96feb58f2e12462ae08191f3cabec64409b3a3" exitCode=0 Oct 29 22:55:36.755893 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:36.755587 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f42f78-cfknt" event={"ID":"58366efd-cd68-466c-bc65-154613ed0ffe","Type":"ContainerDied","Data":"b537fac20fec3d960c92a6feea96feb58f2e12462ae08191f3cabec64409b3a3"} Oct 29 22:55:37.926803 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:37.926774 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f42f78-cfknt" Oct 29 22:55:37.944187 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:37.944160 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58366efd-cd68-466c-bc65-154613ed0ffe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58366efd-cd68-466c-bc65-154613ed0ffe-maas-api-openshift-ca\") pod \"58366efd-cd68-466c-bc65-154613ed0ffe\" (UID: \"58366efd-cd68-466c-bc65-154613ed0ffe\") " Oct 29 22:55:37.944351 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:37.944288 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58366efd-cd68-466c-bc65-154613ed0ffe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58366efd-cd68-466c-bc65-154613ed0ffe-maas-api-xks-ca\") pod \"58366efd-cd68-466c-bc65-154613ed0ffe\" (UID: \"58366efd-cd68-466c-bc65-154613ed0ffe\") " Oct 29 22:55:37.944420 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:37.944390 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58366efd-cd68-466c-bc65-154613ed0ffe-kube-api-access-zdnvd\" (UniqueName: \"kubernetes.io/projected/58366efd-cd68-466c-bc65-154613ed0ffe-kube-api-access-zdnvd\") pod \"58366efd-cd68-466c-bc65-154613ed0ffe\" (UID: \"58366efd-cd68-466c-bc65-154613ed0ffe\") " Oct 29 22:55:37.944567 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:37.944545 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58366efd-cd68-466c-bc65-154613ed0ffe-maas-api-openshift-ca" pod "58366efd-cd68-466c-bc65-154613ed0ffe" (UID: "58366efd-cd68-466c-bc65-154613ed0ffe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:55:37.944854 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:37.944833 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58366efd-cd68-466c-bc65-154613ed0ffe-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:55:37.946534 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:37.946509 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58366efd-cd68-466c-bc65-154613ed0ffe-maas-api-xks-ca" pod "58366efd-cd68-466c-bc65-154613ed0ffe" (UID: "58366efd-cd68-466c-bc65-154613ed0ffe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:55:37.946808 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:37.946785 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58366efd-cd68-466c-bc65-154613ed0ffe-kube-api-access-zdnvd" pod "58366efd-cd68-466c-bc65-154613ed0ffe" (UID: "58366efd-cd68-466c-bc65-154613ed0ffe"). InnerVolumeSpecName "kube-api-access-zdnvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:55:38.007702 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.007671 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh"] Oct 29 22:55:38.008229 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.008192 4821 cpu_manager.go:414] "containerMap: removing container" podUID="58366efd-cd68-466c-bc65-154613ed0ffe" containerName="revoke-keys" Oct 29 22:55:38.008229 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.008228 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58366efd-cd68-466c-bc65-154613ed0ffe" containerName="revoke-keys" Oct 29 22:55:38.008517 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.008383 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="58366efd-cd68-466c-bc65-154613ed0ffe" containerName="revoke-keys" Oct 29 22:55:38.012488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.012469 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.043071 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.043041 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh"] Oct 29 22:55:38.046241 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.046183 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.046358 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.046245 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.046358 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.046298 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.046440 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.046402 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.046440 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.046430 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.046511 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.046472 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.046562 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.046547 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.046602 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.046585 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.046659 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.046645 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksx9f\" (UniqueName: \"kubernetes.io/projected/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-kube-api-access-ksx9f\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.046737 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.046725 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdnvd\" (UniqueName: \"kubernetes.io/projected/58366efd-cd68-466c-bc65-154613ed0ffe-kube-api-access-zdnvd\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:55:38.046783 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.046743 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58366efd-cd68-466c-bc65-154613ed0ffe-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:55:38.147871 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.147835 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksx9f\" (UniqueName: \"kubernetes.io/projected/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-kube-api-access-ksx9f\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.148032 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.147895 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.148032 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.147923 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.148032 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.147958 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.148173 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.148062 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.148173 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.148087 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.148173 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.148122 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.148356 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.148192 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.148356 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.148260 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.148463 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.148357 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.148555 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.148523 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.148691 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.148582 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.148753 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.148715 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.148822 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.148804 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.150735 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.150713 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.150869 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.150851 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.170519 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.170491 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.170687 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.170668 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksx9f\" (UniqueName: \"kubernetes.io/projected/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-kube-api-access-ksx9f\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.327582 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.327514 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:38.662988 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.662965 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh"] Oct 29 22:55:38.663732 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:55:38.663693 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2e7b94_f0cf_424d_a296_c2fe8e60ff01.slice/crio-7a85f7e8a989e2db21566b6efef32b6a27efe7cb04c486a082d343c059371eea WatchSource:0}: Error finding container 7a85f7e8a989e2db21566b6efef32b6a27efe7cb04c486a082d343c059371eea: Status 404 returned error can't find the container with id 7a85f7e8a989e2db21566b6efef32b6a27efe7cb04c486a082d343c059371eea Oct 29 22:55:38.666170 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.666138 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:55:38.666260 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.666228 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:55:38.666307 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.666262 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:55:38.771875 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.771833 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f42f78-cfknt" Oct 29 22:55:38.771875 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.771864 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f42f78-cfknt" event={"ID":"58366efd-cd68-466c-bc65-154613ed0ffe","Type":"ContainerDied","Data":"43280f9dd02178f06f0886cc39a2880b9a4dccc6cbc1a4c2db5f50f29e30ad34"} Oct 29 22:55:38.772121 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.771912 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43280f9dd02178f06f0886cc39a2880b9a4dccc6cbc1a4c2db5f50f29e30ad34" Oct 29 22:55:38.773876 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.773842 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" event={"ID":"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01","Type":"ContainerStarted","Data":"a8c7cdf4c3a29e9f2627730816b381e54a3446275af17a1f0f7fc7f0d4ffbf31"} Oct 29 22:55:38.774022 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.773892 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" event={"ID":"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01","Type":"ContainerStarted","Data":"7a85f7e8a989e2db21566b6efef32b6a27efe7cb04c486a082d343c059371eea"} Oct 29 22:55:38.820209 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:38.820159 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" podStartSLOduration=1.820145218 podStartE2EDuration="1.820145218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:38.817656191 +0000 UTC m=+792.758204102" watchObservedRunningTime="2026-10-29 22:55:38.820145218 +0000 UTC m=+792.760693128" Oct 29 22:55:39.336225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:39.335573 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:39.339358 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:39.339321 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:55:39.339524 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:39.339404 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" podUID="0c2e7b94-f0cf-424d-a296-c2fe8e60ff01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 29 22:55:40.338227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:40.328543 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:55:40.338227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:40.328620 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" podUID="0c2e7b94-f0cf-424d-a296-c2fe8e60ff01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 29 22:55:41.335226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:41.334583 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:55:41.335226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:41.334647 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" podUID="0c2e7b94-f0cf-424d-a296-c2fe8e60ff01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 29 22:55:42.336227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:42.332588 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:55:42.336227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:42.332662 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" podUID="0c2e7b94-f0cf-424d-a296-c2fe8e60ff01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 29 22:55:43.333228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:43.331583 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:55:43.333228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:43.331668 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" podUID="0c2e7b94-f0cf-424d-a296-c2fe8e60ff01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 29 22:55:44.332227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:44.331564 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:55:44.332227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:44.331621 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" podUID="0c2e7b94-f0cf-424d-a296-c2fe8e60ff01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 29 22:55:45.328765 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:45.328722 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:55:45.328959 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:45.328790 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" podUID="0c2e7b94-f0cf-424d-a296-c2fe8e60ff01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 29 22:55:46.332160 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:46.332131 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:46.332605 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:46.332414 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:46.333142 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:46.333122 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 22:55:47.265079 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:47.265041 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v"] Oct 29 22:55:47.269837 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:47.269813 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:55:47.301088 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:47.301047 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e131af\"/\"distinct-2-w1-e131af-kserve-self-signed-certs\"" Oct 29 22:55:47.304377 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:47.304350 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v"] Oct 29 22:55:47.392981 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:47.392946 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-model-cache\") pod \"distinct-2-w1-e131af-kserve-7486ff57db-bbv9v\" (UID: \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:55:47.393777 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:47.393746 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-tmp-dir\") pod \"distinct-2-w1-e131af-kserve-7486ff57db-bbv9v\" (UID: \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:55:47.393937 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:47.393901 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqrhk\" (UniqueName: \"kubernetes.io/projected/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-kube-api-access-mqrhk\") pod \"distinct-2-w1-e131af-kserve-7486ff57db-bbv9v\" (UID: \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:55:47.394010 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:47.393992 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-dshm\") pod \"distinct-2-w1-e131af-kserve-7486ff57db-bbv9v\" (UID: \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:55:47.394131 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:47.394113 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-tls-certs\") pod \"distinct-2-w1-e131af-kserve-7486ff57db-bbv9v\" (UID: \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:55:47.394235 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:47.394185 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-home\") pod \"distinct-2-w1-e131af-kserve-7486ff57db-bbv9v\" (UID: \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:55:47.504220 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:47.496960 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-dshm\") pod \"distinct-2-w1-e131af-kserve-7486ff57db-bbv9v\" (UID: \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:55:47.504220 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:47.497109 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-tls-certs\") pod \"distinct-2-w1-e131af-kserve-7486ff57db-bbv9v\" (UID: \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:55:47.504220 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:47.497173 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-home\") pod \"distinct-2-w1-e131af-kserve-7486ff57db-bbv9v\" (UID: \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:55:47.504220 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:47.497253 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-model-cache\") pod \"distinct-2-w1-e131af-kserve-7486ff57db-bbv9v\" (UID: \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:55:47.504220 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:47.497300 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-tmp-dir\") pod \"distinct-2-w1-e131af-kserve-7486ff57db-bbv9v\" (UID: \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:55:47.504220 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:47.497372 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqrhk\" (UniqueName: \"kubernetes.io/projected/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-kube-api-access-mqrhk\") pod \"distinct-2-w1-e131af-kserve-7486ff57db-bbv9v\" (UID: \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:55:47.508683 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:47.508653 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-home\") pod \"distinct-2-w1-e131af-kserve-7486ff57db-bbv9v\" (UID: \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:55:47.508857 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:47.508529 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-model-cache\") pod \"distinct-2-w1-e131af-kserve-7486ff57db-bbv9v\" (UID: \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:55:47.508857 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:47.508753 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-tmp-dir\") pod \"distinct-2-w1-e131af-kserve-7486ff57db-bbv9v\" (UID: \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:55:47.512844 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:47.512815 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-tls-certs\") pod \"distinct-2-w1-e131af-kserve-7486ff57db-bbv9v\" (UID: \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:55:47.526739 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:47.526659 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-dshm\") pod \"distinct-2-w1-e131af-kserve-7486ff57db-bbv9v\" (UID: \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:55:47.565046 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:47.565002 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqrhk\" (UniqueName: \"kubernetes.io/projected/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-kube-api-access-mqrhk\") pod \"distinct-2-w1-e131af-kserve-7486ff57db-bbv9v\" (UID: \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\") " pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:55:47.585791 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:47.585615 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:55:47.913049 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:47.913015 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v"] Oct 29 22:55:47.916431 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:55:47.916400 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59c07a4a_dee9_4e00_a10b_eee2f9160fd1.slice/crio-7ae1996dd5e83892b308ff7cca7c6137f03742ac0e55a784422f1785b804d191 WatchSource:0}: Error finding container 7ae1996dd5e83892b308ff7cca7c6137f03742ac0e55a784422f1785b804d191: Status 404 returned error can't find the container with id 7ae1996dd5e83892b308ff7cca7c6137f03742ac0e55a784422f1785b804d191 Oct 29 22:55:48.848609 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:48.848572 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" event={"ID":"59c07a4a-dee9-4e00-a10b-eee2f9160fd1","Type":"ContainerStarted","Data":"0f2658ce58eb7f018465f05bf9df5593e4b69dc85fd69d3ab6003ffcaf2aab04"} Oct 29 22:55:48.849086 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:48.848625 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" event={"ID":"59c07a4a-dee9-4e00-a10b-eee2f9160fd1","Type":"ContainerStarted","Data":"7ae1996dd5e83892b308ff7cca7c6137f03742ac0e55a784422f1785b804d191"} Oct 29 22:55:48.890828 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:48.890777 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podStartSLOduration=1.890763744 podStartE2EDuration="1.890763744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:48.889329071 +0000 UTC m=+802.829876983" watchObservedRunningTime="2026-10-29 22:55:48.890763744 +0000 UTC m=+802.831311690" Oct 29 22:55:49.538984 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.538951 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8"] Oct 29 22:55:49.542748 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.542726 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.587032 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.586988 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8"] Oct 29 22:55:49.587234 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.587117 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:55:49.626872 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.626838 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.627057 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.627023 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.627180 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.627089 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.627180 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.627140 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.627329 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.627258 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27rj6\" (UniqueName: \"kubernetes.io/projected/e438de1e-67a0-44a8-8ecd-e38ca050a541-kube-api-access-27rj6\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.627329 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.627316 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.627416 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.627350 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.627462 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.627450 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e438de1e-67a0-44a8-8ecd-e38ca050a541-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.627529 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.627504 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.728648 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.728611 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.728847 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.728669 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.728847 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.728700 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.728847 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.728726 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.728847 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.728795 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27rj6\" (UniqueName: \"kubernetes.io/projected/e438de1e-67a0-44a8-8ecd-e38ca050a541-kube-api-access-27rj6\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.728847 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.728843 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.729101 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.728878 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.729101 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.728934 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e438de1e-67a0-44a8-8ecd-e38ca050a541-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.729101 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.728993 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.729222 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.729107 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.729527 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.729495 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.729657 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.729548 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.729657 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.729595 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.729839 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.729819 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e438de1e-67a0-44a8-8ecd-e38ca050a541-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.731470 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.731450 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.731470 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.731463 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.756322 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.753182 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.756322 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.753586 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27rj6\" (UniqueName: \"kubernetes.io/projected/e438de1e-67a0-44a8-8ecd-e38ca050a541-kube-api-access-27rj6\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:49.857095 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:49.857066 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:50.215813 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:50.215784 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8"] Oct 29 22:55:50.217286 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:55:50.217257 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode438de1e_67a0_44a8_8ecd_e38ca050a541.slice/crio-0cb1439dc607e0e7e09baffb5c67715462030ff94d21eaa92e8a38d0cd39ec9f WatchSource:0}: Error finding container 0cb1439dc607e0e7e09baffb5c67715462030ff94d21eaa92e8a38d0cd39ec9f: Status 404 returned error can't find the container with id 0cb1439dc607e0e7e09baffb5c67715462030ff94d21eaa92e8a38d0cd39ec9f Oct 29 22:55:50.220648 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:50.220614 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:55:50.220750 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:50.220682 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:55:50.220750 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:50.220715 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:55:50.864271 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:50.864234 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" event={"ID":"e438de1e-67a0-44a8-8ecd-e38ca050a541","Type":"ContainerStarted","Data":"cba975f80f91d84537dda3e737344dff853e8031ec6ccae24e3efe6f9df0e8b3"} Oct 29 22:55:50.864717 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:50.864297 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" event={"ID":"e438de1e-67a0-44a8-8ecd-e38ca050a541","Type":"ContainerStarted","Data":"0cb1439dc607e0e7e09baffb5c67715462030ff94d21eaa92e8a38d0cd39ec9f"} Oct 29 22:55:50.908638 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:50.908577 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" podStartSLOduration=1.9085588869999999 podStartE2EDuration="1.908558887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:50.907811337 +0000 UTC m=+804.848359246" watchObservedRunningTime="2026-10-29 22:55:50.908558887 +0000 UTC m=+804.849106800" Oct 29 22:55:51.075316 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.075283 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn"] Oct 29 22:55:51.079414 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.079393 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.109468 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.109436 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn"] Oct 29 22:55:51.263588 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.263488 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.263769 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.263640 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c0101f-490c-470e-b945-e3bb7c86d2c6-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.263769 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.263676 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctv5z\" (UniqueName: \"kubernetes.io/projected/62c0101f-490c-470e-b945-e3bb7c86d2c6-kube-api-access-ctv5z\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.263769 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.263760 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.263928 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.263904 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.264018 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.264003 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.264057 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.264038 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.264232 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.264193 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.264388 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.264370 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.365262 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.365183 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.365443 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.365283 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c0101f-490c-470e-b945-e3bb7c86d2c6-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.365443 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.365303 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctv5z\" (UniqueName: \"kubernetes.io/projected/62c0101f-490c-470e-b945-e3bb7c86d2c6-kube-api-access-ctv5z\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.365443 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.365338 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.365443 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.365364 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.365648 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.365474 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.365648 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.365504 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.365648 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.365562 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.365648 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.365620 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.365850 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.365617 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.365850 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.365746 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.365850 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.365805 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.365989 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.365876 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.366046 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.366027 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c0101f-490c-470e-b945-e3bb7c86d2c6-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.368065 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.368035 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.368166 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.368111 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.388558 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.388532 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.388876 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.388855 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctv5z\" (UniqueName: \"kubernetes.io/projected/62c0101f-490c-470e-b945-e3bb7c86d2c6-kube-api-access-ctv5z\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.394232 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.394194 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:51.746665 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:55:51.746637 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c0101f_490c_470e_b945_e3bb7c86d2c6.slice/crio-de32924776c41532e8c864cf101ec71c33699162edce8043b3fb6a11e0edd862 WatchSource:0}: Error finding container de32924776c41532e8c864cf101ec71c33699162edce8043b3fb6a11e0edd862: Status 404 returned error can't find the container with id de32924776c41532e8c864cf101ec71c33699162edce8043b3fb6a11e0edd862 Oct 29 22:55:51.748621 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.748480 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn"] Oct 29 22:55:51.750358 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.750326 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:55:51.750469 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.750402 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:55:51.750469 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.750446 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:55:51.857731 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.857700 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:55:51.859397 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.859368 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:55:51.859557 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.859414 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" podUID="e438de1e-67a0-44a8-8ecd-e38ca050a541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 29 22:55:51.873754 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.873715 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" event={"ID":"62c0101f-490c-470e-b945-e3bb7c86d2c6","Type":"ContainerStarted","Data":"5a94f480d920d388d547679f958ac5acc83b198e28d5b7c50c08f7c7211b1975"} Oct 29 22:55:51.874191 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.873781 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" event={"ID":"62c0101f-490c-470e-b945-e3bb7c86d2c6","Type":"ContainerStarted","Data":"de32924776c41532e8c864cf101ec71c33699162edce8043b3fb6a11e0edd862"} Oct 29 22:55:51.942223 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:51.941429 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" podStartSLOduration=0.941409974 podStartE2EDuration="941.409974ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:51.940812074 +0000 UTC m=+805.881359985" watchObservedRunningTime="2026-10-29 22:55:51.941409974 +0000 UTC m=+805.881957886" Oct 29 22:55:52.395134 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:52.395103 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:55:52.396933 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:52.396907 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:55:52.397060 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:52.396956 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" podUID="62c0101f-490c-470e-b945-e3bb7c86d2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 29 22:55:52.862316 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:52.858011 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:55:52.862316 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:52.858074 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" podUID="e438de1e-67a0-44a8-8ecd-e38ca050a541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 29 22:55:53.066227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.065463 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774"] Oct 29 22:55:53.073354 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.073327 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.087469 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.087436 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.087730 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.087707 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.087949 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.087930 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.088099 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.088081 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.088302 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.088284 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.088450 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.088432 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.088567 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.088551 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae79f378-d055-4e08-bd3a-1e75fcc38882-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.088682 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.088667 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.088888 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.088869 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7zkz\" (UniqueName: \"kubernetes.io/projected/ae79f378-d055-4e08-bd3a-1e75fcc38882-kube-api-access-t7zkz\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.094299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.094165 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774"] Oct 29 22:55:53.189935 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.189800 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.189935 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.189857 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.189935 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.189878 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae79f378-d055-4e08-bd3a-1e75fcc38882-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.189935 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.189918 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.190261 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.190072 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7zkz\" (UniqueName: \"kubernetes.io/projected/ae79f378-d055-4e08-bd3a-1e75fcc38882-kube-api-access-t7zkz\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.190261 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.190105 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.190261 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.190160 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.190426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.190267 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.190426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.190315 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.190426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.190321 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.190662 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.190443 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.190662 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.190543 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.190779 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.190694 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.191078 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.191059 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae79f378-d055-4e08-bd3a-1e75fcc38882-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.192587 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.192561 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.196005 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.195596 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.213128 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.213098 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7zkz\" (UniqueName: \"kubernetes.io/projected/ae79f378-d055-4e08-bd3a-1e75fcc38882-kube-api-access-t7zkz\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.232226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.215620 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.393579 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.393546 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:53.394872 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.394846 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:55:53.394932 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.394908 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" podUID="62c0101f-490c-470e-b945-e3bb7c86d2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 29 22:55:53.772561 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.772529 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774"] Oct 29 22:55:53.773653 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:55:53.773627 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae79f378_d055_4e08_bd3a_1e75fcc38882.slice/crio-bb96fcb876c295bf73155ecf548fc0d6a4391570a24bfda35d9b31c8dced51db WatchSource:0}: Error finding container bb96fcb876c295bf73155ecf548fc0d6a4391570a24bfda35d9b31c8dced51db: Status 404 returned error can't find the container with id bb96fcb876c295bf73155ecf548fc0d6a4391570a24bfda35d9b31c8dced51db Oct 29 22:55:53.776401 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.776372 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:55:53.776461 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.776446 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:55:53.776503 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.776478 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:55:53.866316 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.863101 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:55:53.866316 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.863446 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" podUID="e438de1e-67a0-44a8-8ecd-e38ca050a541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 29 22:55:53.904273 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:53.903678 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" event={"ID":"ae79f378-d055-4e08-bd3a-1e75fcc38882","Type":"ContainerStarted","Data":"bb96fcb876c295bf73155ecf548fc0d6a4391570a24bfda35d9b31c8dced51db"} Oct 29 22:55:53.908438 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:55:53.907732 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:55:54.395737 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.395703 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:55:54.396312 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.396287 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" podUID="62c0101f-490c-470e-b945-e3bb7c86d2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 29 22:55:54.399305 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.399282 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb"] Oct 29 22:55:54.403462 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.403444 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.404659 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.404636 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istiod-ca-cert\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.404769 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.404679 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-workload-certs\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.404769 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.404696 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-data\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.404769 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.404715 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-credential-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.404769 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.404732 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2qgm\" (UniqueName: \"kubernetes.io/projected/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-kube-api-access-c2qgm\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.404769 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.404752 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-podinfo\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.405037 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.404794 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-workload-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.405037 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.404852 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-envoy\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.405037 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.404873 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-token\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.452226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.449791 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb"] Oct 29 22:55:54.505882 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.505748 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-token\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.505882 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.505834 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istiod-ca-cert\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.505882 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.505868 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-workload-certs\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.505882 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.505884 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-data\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.506258 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.505903 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-credential-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.506258 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.505919 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2qgm\" (UniqueName: \"kubernetes.io/projected/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-kube-api-access-c2qgm\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.506258 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.505943 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-podinfo\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.506258 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.505983 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-workload-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.506258 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.506026 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-envoy\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.506691 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.506667 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-data\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.507378 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.507348 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-workload-certs\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.507652 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.507631 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-workload-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.507719 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.507679 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-credential-socket\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.508758 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.508734 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istiod-ca-cert\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.510229 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.509337 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-envoy\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.510229 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.510072 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-podinfo\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.579246 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.577408 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-token\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.580399 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.580357 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2qgm\" (UniqueName: \"kubernetes.io/projected/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-kube-api-access-c2qgm\") pod \"e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.755281 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.754471 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:54.867137 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.863634 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:55:54.867137 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.863688 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" podUID="e438de1e-67a0-44a8-8ecd-e38ca050a541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 29 22:55:54.999083 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:54.997987 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" event={"ID":"ae79f378-d055-4e08-bd3a-1e75fcc38882","Type":"ContainerStarted","Data":"f13f2b0b590d9d8bc228994b0979fe93cd11c43b58855606a64cb24fe97f5f48"} Oct 29 22:55:55.100285 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:55.098115 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" podStartSLOduration=2.098096477 podStartE2EDuration="2.098096477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:55.097901811 +0000 UTC m=+809.038449722" watchObservedRunningTime="2026-10-29 22:55:55.098096477 +0000 UTC m=+809.038644389" Oct 29 22:55:55.197403 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:55.197369 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb"] Oct 29 22:55:55.205603 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:55.205564 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:55:55.205675 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:55.205658 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:55:55.205721 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:55.205711 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:55:55.398703 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:55.398516 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:55:55.399259 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:55.398836 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:55:55.399259 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:55.398887 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" podUID="62c0101f-490c-470e-b945-e3bb7c86d2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 29 22:55:55.402934 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:55.402295 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 29 22:55:55.402934 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:55.402341 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" podUID="ae79f378-d055-4e08-bd3a-1e75fcc38882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 29 22:55:55.878059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:55.878018 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:55:55.878441 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:55.878406 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" podUID="e438de1e-67a0-44a8-8ecd-e38ca050a541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 29 22:55:56.139445 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:56.139342 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" event={"ID":"fc867e1c-3627-44c9-bc0a-94765ae3d2d7","Type":"ContainerStarted","Data":"4ec8a37dd43adcd287b69538870ca8c96325fe5f7bfe7f135e0b221d6b91bbbf"} Oct 29 22:55:56.139738 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:56.139715 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" event={"ID":"fc867e1c-3627-44c9-bc0a-94765ae3d2d7","Type":"ContainerStarted","Data":"6697834f3994647c2c438c153da666d81cd7b59f27811a0e555e65a0068eb6cb"} Oct 29 22:55:56.206508 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:56.206432 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" podStartSLOduration=2.206413048 podStartE2EDuration="2.206413048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:56.20355599 +0000 UTC m=+810.144103902" watchObservedRunningTime="2026-10-29 22:55:56.206413048 +0000 UTC m=+810.146960961" Oct 29 22:55:56.395285 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:56.394591 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 29 22:55:56.395285 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:56.394650 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" podUID="ae79f378-d055-4e08-bd3a-1e75fcc38882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 29 22:55:56.395548 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:56.395319 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:55:56.395548 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:56.395360 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" podUID="62c0101f-490c-470e-b945-e3bb7c86d2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 29 22:55:56.758276 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:56.757862 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:55:56.780631 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:56.770680 4821 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:55:56.780631 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:56.770737 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" podUID="fc867e1c-3627-44c9-bc0a-94765ae3d2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 29 22:55:56.864672 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:56.864636 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:55:56.881387 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:56.881342 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" podUID="e438de1e-67a0-44a8-8ecd-e38ca050a541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 29 22:55:57.398736 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:57.398701 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:55:57.399069 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:57.399040 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" podUID="62c0101f-490c-470e-b945-e3bb7c86d2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 29 22:55:57.400129 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:57.400091 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 29 22:55:57.400343 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:57.400322 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" podUID="ae79f378-d055-4e08-bd3a-1e75fcc38882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 29 22:55:57.589542 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:57.589510 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:55:57.612999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:57.612968 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:55:57.762281 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:57.761588 4821 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:55:57.762281 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:57.761647 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" podUID="fc867e1c-3627-44c9-bc0a-94765ae3d2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 29 22:55:57.859225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:57.858592 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:55:57.859225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:57.858655 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" podUID="e438de1e-67a0-44a8-8ecd-e38ca050a541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 29 22:55:58.006279 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:58.005291 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:58.152339 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:58.152297 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:58.316224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:58.315346 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:58.383345 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:58.383287 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:58.407681 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:58.407589 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:55:58.407929 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:58.407901 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" podUID="62c0101f-490c-470e-b945-e3bb7c86d2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 29 22:55:58.408990 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:58.408965 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 29 22:55:58.409172 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:58.409149 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" podUID="ae79f378-d055-4e08-bd3a-1e75fcc38882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 29 22:55:58.460347 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:58.460298 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:58.649225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:58.631294 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 29 22:55:58.780279 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:58.773643 4821 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:55:58.780279 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:58.773725 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" podUID="fc867e1c-3627-44c9-bc0a-94765ae3d2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 29 22:55:58.938232 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:58.917349 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:58.938232 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:58.919604 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:59.010227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:59.006892 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:59.010227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:59.007692 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:55:59.017505 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:59.017465 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:59.167225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:59.156784 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 29 22:55:59.180418 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:59.180386 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:55:59.242516 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:59.222587 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:59.272548 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:59.272323 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:59.345228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:59.275408 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:59.345228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:59.293105 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:59.345228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:59.323932 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 29 22:55:59.345228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:59.324342 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:55:59.406483 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:59.406441 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:55:59.406794 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:59.406758 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" podUID="62c0101f-490c-470e-b945-e3bb7c86d2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 29 22:55:59.422284 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:59.409730 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 29 22:55:59.514502 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:59.414228 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:55:59.525385 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:59.515595 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" podUID="ae79f378-d055-4e08-bd3a-1e75fcc38882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 29 22:55:59.525691 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:59.413845 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:59.526914 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:59.526890 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:55:59.785924 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:59.785833 4821 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:55:59.814471 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:59.814429 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" podUID="fc867e1c-3627-44c9-bc0a-94765ae3d2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 29 22:55:59.905955 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:59.905323 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:55:59.905955 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:55:59.905401 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" podUID="e438de1e-67a0-44a8-8ecd-e38ca050a541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:00.015245 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:00.008636 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:00.121414 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:00.121368 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:56:00.185227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:00.171315 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:00.272384 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:00.272333 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:00.281401 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:00.280517 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 29 22:56:00.281401 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:00.280849 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:56:00.362229 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:00.348701 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 29 22:56:00.362229 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:00.351264 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:00.362229 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:00.355289 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:56:00.428164 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:00.428063 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:56:00.444068 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:00.428424 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" podUID="62c0101f-490c-470e-b945-e3bb7c86d2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 29 22:56:00.453384 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:00.430433 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 29 22:56:00.453632 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:00.453608 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" podUID="ae79f378-d055-4e08-bd3a-1e75fcc38882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 29 22:56:00.455867 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:00.455844 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:56:00.456783 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:00.456768 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:56:00.461449 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:00.461425 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 22:56:00.522983 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:00.522955 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:56:00.527304 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:00.527265 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:00.736042 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:00.735950 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:56:00.756741 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:00.756648 4821 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:56:00.757042 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:00.757010 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" podUID="fc867e1c-3627-44c9-bc0a-94765ae3d2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 29 22:56:00.948356 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:00.948303 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:01.072286 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:01.055938 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:01.270761 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:01.270706 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:01.359292 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:01.357072 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 29 22:56:01.417120 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:01.416944 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 29 22:56:01.418071 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:01.418021 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" podUID="ae79f378-d055-4e08-bd3a-1e75fcc38882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 29 22:56:01.445662 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:01.445623 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:01.533347 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:01.533299 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:01.616414 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:01.616304 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:01.640464 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:01.640414 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 29 22:56:01.758629 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:01.758592 4821 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:56:01.758906 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:01.758875 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" podUID="fc867e1c-3627-44c9-bc0a-94765ae3d2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 29 22:56:01.910326 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:01.907304 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:01.911785 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:01.911711 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 29 22:56:02.056247 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.026308 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 29 22:56:02.084229 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.073357 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:02.084938 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.084908 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:56:02.301279 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.277295 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 29 22:56:02.301279 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.277682 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 29 22:56:02.301279 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.279042 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:02.371340 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.371293 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:02.371986 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.371954 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:02.393924 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.393890 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:56:02.394189 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.394155 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" podUID="0c2e7b94-f0cf-424d-a296-c2fe8e60ff01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:02.407491 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.407374 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 29 22:56:02.407491 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.407430 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" podUID="ae79f378-d055-4e08-bd3a-1e75fcc38882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 29 22:56:02.467225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.462985 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:02.467225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.462955 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:56:02.467225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.463087 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" podUID="62c0101f-490c-470e-b945-e3bb7c86d2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:02.502227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.487157 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:56:02.597762 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.597665 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:02.683241 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.621793 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:02.683241 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.622825 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:56:02.683241 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.644099 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 29 22:56:02.683241 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.644520 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:56:02.890226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.889609 4821 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:56:02.890226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.889671 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" podUID="fc867e1c-3627-44c9-bc0a-94765ae3d2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 29 22:56:02.929227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.926748 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 22:56:02.929227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:02.927095 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:56:03.042184 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.042139 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:03.042867 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.042842 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:56:03.090499 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.090460 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:03.284286 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.283282 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:03.284286 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.283807 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 29 22:56:03.526222 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.522348 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 29 22:56:03.526222 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.522483 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 29 22:56:03.542173 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.542074 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:56:03.552232 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.544326 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:03.552232 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.544807 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:56:03.552232 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.544894 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:03.552232 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.545966 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:56:03.580228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.562398 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:03.580228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.565456 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:56:03.580228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.565517 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" podUID="62c0101f-490c-470e-b945-e3bb7c86d2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:03.580228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.578732 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 29 22:56:03.580228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.578794 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" podUID="ae79f378-d055-4e08-bd3a-1e75fcc38882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 29 22:56:03.622230 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.611606 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 29 22:56:03.622230 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.612087 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sn4mm" podUID="bdeff50a-75a8-4272-9120-47c753cdcb5f" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.23:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:03.666042 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.628240 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:03.666042 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.628365 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:03.867290 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.827740 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:56:04.005743 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.005702 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:56:04.035972 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.884918 4821 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:56:04.117227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.071689 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" podUID="fc867e1c-3627-44c9-bc0a-94765ae3d2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 29 22:56:04.266578 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.958586 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sn4mm" podUID="bdeff50a-75a8-4272-9120-47c753cdcb5f" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.23:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:04.293099 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.864485 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 29 22:56:04.293448 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.036888 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 29 22:56:04.293729 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.067118 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 22:56:04.293993 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:03.895305 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:04.294232 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.172109 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 22:56:04.294481 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.175142 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 29 22:56:04.294763 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.176317 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:56:04.414233 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.372404 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:04.453784 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.393928 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 29 22:56:04.454131 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.454058 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" podUID="ae79f378-d055-4e08-bd3a-1e75fcc38882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 29 22:56:04.454617 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.394809 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:04.542464 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.542359 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 29 22:56:04.542799 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.542772 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 29 22:56:04.559597 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.556278 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:04.559597 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.557304 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:04.559597 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.557666 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:56:04.568233 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.564512 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:04.579383 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.576441 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:04.595861 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.595829 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:56:04.707344 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.707278 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:04.708623 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.708586 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:04.765703 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.765663 4821 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:56:04.769111 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.769073 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" podUID="fc867e1c-3627-44c9-bc0a-94765ae3d2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 29 22:56:04.871288 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.850695 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:04.887336 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.887293 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:04.972236 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:04.970558 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:56:05.298403 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:05.298292 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:05.300421 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:05.300388 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:05.302063 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:05.302038 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 22:56:05.396572 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:05.396540 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 29 22:56:05.396865 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:05.396821 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" podUID="ae79f378-d055-4e08-bd3a-1e75fcc38882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 29 22:56:05.469760 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:05.469704 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:05.554279 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:05.543273 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:05.569262 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:05.565323 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:05.569262 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:05.565723 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 29 22:56:05.569262 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:05.567423 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:05.601316 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:05.594022 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:05.716240 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:05.708302 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:05.763239 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:05.757393 4821 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:56:05.763239 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:05.757456 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" podUID="fc867e1c-3627-44c9-bc0a-94765ae3d2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 29 22:56:05.868982 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:05.866300 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:06.123560 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.123464 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:56:06.337787 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.337747 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 29 22:56:06.338577 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.338542 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:06.339160 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.339081 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:56:06.475546 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.470407 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 29 22:56:06.569223 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.557430 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:06.569563 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.569530 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:06.570104 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.570067 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 29 22:56:06.576293 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.574582 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:06.610257 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.601281 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ls7vc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:56:06.610257 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.601353 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" podUID="2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:06.631700 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.631666 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:06.644403 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.640562 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:56:06.644403 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.642182 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:56:06.698359 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.698329 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:56:06.709285 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.709152 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:06.739627 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.738597 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:56:06.763403 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.763368 4821 patch_prober.go:29] interesting pod/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:56:06.763685 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.763656 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" podUID="fc867e1c-3627-44c9-bc0a-94765ae3d2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 29 22:56:06.798624 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.798597 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:56:06.921789 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.921731 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 29 22:56:06.928496 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.928470 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:56:06.929962 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.929944 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:56:06.942121 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.940710 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:56:06.964405 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.964135 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:56:06.994796 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.993617 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:56:06.996642 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.996615 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 22:56:06.998326 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:06.998307 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:56:07.000088 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:07.000023 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:56:08.233487 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:08.233454 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:56:08.234141 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:08.234115 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:56:08.235110 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:08.235090 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 22:56:08.347766 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:08.345794 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw"] Oct 29 22:56:08.352461 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:08.351381 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw" Oct 29 22:56:08.369371 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:08.369293 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw"] Oct 29 22:56:08.426910 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:08.426871 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e4a049\"" Oct 29 22:56:08.427104 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:08.427036 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e4a049-dockercfg-tvj2f\"" Oct 29 22:56:08.531578 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:08.530947 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfzcm\" (UniqueName: \"kubernetes.io/projected/8e11dce4-40a9-47bf-a689-b3b81d86417b-kube-api-access-dfzcm\") pod \"payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw\" (UID: \"8e11dce4-40a9-47bf-a689-b3b81d86417b\") " pod="openshift-ingress/payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw" Oct 29 22:56:08.531578 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:08.531121 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e11dce4-40a9-47bf-a689-b3b81d86417b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw\" (UID: \"8e11dce4-40a9-47bf-a689-b3b81d86417b\") " pod="openshift-ingress/payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw" Oct 29 22:56:08.632988 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:08.632941 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfzcm\" (UniqueName: \"kubernetes.io/projected/8e11dce4-40a9-47bf-a689-b3b81d86417b-kube-api-access-dfzcm\") pod \"payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw\" (UID: \"8e11dce4-40a9-47bf-a689-b3b81d86417b\") " pod="openshift-ingress/payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw" Oct 29 22:56:08.633252 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:08.633231 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e11dce4-40a9-47bf-a689-b3b81d86417b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw\" (UID: \"8e11dce4-40a9-47bf-a689-b3b81d86417b\") " pod="openshift-ingress/payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw" Oct 29 22:56:08.634363 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:08.634336 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e11dce4-40a9-47bf-a689-b3b81d86417b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw\" (UID: \"8e11dce4-40a9-47bf-a689-b3b81d86417b\") " pod="openshift-ingress/payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw" Oct 29 22:56:08.656194 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:08.656161 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfzcm\" (UniqueName: \"kubernetes.io/projected/8e11dce4-40a9-47bf-a689-b3b81d86417b-kube-api-access-dfzcm\") pod \"payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw\" (UID: \"8e11dce4-40a9-47bf-a689-b3b81d86417b\") " pod="openshift-ingress/payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw" Oct 29 22:56:08.678104 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:08.678038 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw" Oct 29 22:56:09.077279 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:09.076696 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw"] Oct 29 22:56:09.447924 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:09.447141 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw" event={"ID":"8e11dce4-40a9-47bf-a689-b3b81d86417b","Type":"ContainerStarted","Data":"507d523db49884bd713f2bc7b413002af7a66722615927bfedec1469632d0172"} Oct 29 22:56:09.447924 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:09.447249 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw" event={"ID":"8e11dce4-40a9-47bf-a689-b3b81d86417b","Type":"ContainerStarted","Data":"a449827b764cd8c22e5016a3459e8762b985015d3e21817dbaf5113c70ea1b5e"} Oct 29 22:56:09.447924 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:09.447600 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw" Oct 29 22:56:09.487808 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:09.487759 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw" podStartSLOduration=1.487745127 podStartE2EDuration="1.487745127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:09.486287025 +0000 UTC m=+823.426834936" watchObservedRunningTime="2026-10-29 22:56:09.487745127 +0000 UTC m=+823.428293037" Oct 29 22:56:12.193742 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.193708 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb"] Oct 29 22:56:12.197615 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.197594 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.227732 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.227706 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb"] Oct 29 22:56:12.302985 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.302940 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.303186 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.303001 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.303186 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.303135 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.303186 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.303174 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrf9c\" (UniqueName: \"kubernetes.io/projected/b1fb295f-b287-441f-a057-efaf16a8c8cc-kube-api-access-vrf9c\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.303352 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.303309 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.303447 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.303431 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.303489 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.303474 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1fb295f-b287-441f-a057-efaf16a8c8cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.303624 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.303608 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.303690 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.303645 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.405080 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.405040 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.405266 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.405097 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.405266 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.405174 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.405266 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.405234 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.405439 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.405379 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.405504 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.405430 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrf9c\" (UniqueName: \"kubernetes.io/projected/b1fb295f-b287-441f-a057-efaf16a8c8cc-kube-api-access-vrf9c\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.405557 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.405510 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.405557 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.405429 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.405657 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.405630 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.405657 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.405647 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.405767 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.405686 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1fb295f-b287-441f-a057-efaf16a8c8cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.405851 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.405824 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.406017 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.405995 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.406357 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.406340 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1fb295f-b287-441f-a057-efaf16a8c8cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.408145 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.408116 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.408287 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.408119 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.428990 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.428962 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.429102 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.429041 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrf9c\" (UniqueName: \"kubernetes.io/projected/b1fb295f-b287-441f-a057-efaf16a8c8cc-kube-api-access-vrf9c\") pod \"maas-default-gateway-openshift-default-6db586dd6c-2hddb\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.511901 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.511818 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:12.836033 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.836001 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb"] Oct 29 22:56:12.836670 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:56:12.836635 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1fb295f_b287_441f_a057_efaf16a8c8cc.slice/crio-a558cde18403b9b3309a8c0cf2273d13161438014f514dd52314847245462238 WatchSource:0}: Error finding container a558cde18403b9b3309a8c0cf2273d13161438014f514dd52314847245462238: Status 404 returned error can't find the container with id a558cde18403b9b3309a8c0cf2273d13161438014f514dd52314847245462238 Oct 29 22:56:12.839215 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.839166 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:56:12.839311 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.839261 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:56:12.839311 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:12.839306 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:56:13.476039 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:13.476000 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" event={"ID":"b1fb295f-b287-441f-a057-efaf16a8c8cc","Type":"ContainerStarted","Data":"0c12224e7da4e3f0c66a9c4d8dd198c68acfd2cfd73706e843d500fffb69b16e"} Oct 29 22:56:13.476550 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:13.476070 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" event={"ID":"b1fb295f-b287-441f-a057-efaf16a8c8cc","Type":"ContainerStarted","Data":"a558cde18403b9b3309a8c0cf2273d13161438014f514dd52314847245462238"} Oct 29 22:56:13.512149 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:13.512126 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:13.513566 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:13.513538 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2hddb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:56:13.513699 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:13.513580 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" podUID="b1fb295f-b287-441f-a057-efaf16a8c8cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 29 22:56:13.545450 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:13.545401 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" podStartSLOduration=1.545384739 podStartE2EDuration="1.545384739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:13.533778789 +0000 UTC m=+827.474326700" watchObservedRunningTime="2026-10-29 22:56:13.545384739 +0000 UTC m=+827.485932649" Oct 29 22:56:14.513060 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:14.513023 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2hddb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:56:14.513483 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:14.513074 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" podUID="b1fb295f-b287-441f-a057-efaf16a8c8cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 29 22:56:15.512962 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:15.512928 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2hddb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:56:15.513124 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:15.512976 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" podUID="b1fb295f-b287-441f-a057-efaf16a8c8cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 29 22:56:16.512914 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:16.512880 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2hddb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:56:16.513082 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:16.512932 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" podUID="b1fb295f-b287-441f-a057-efaf16a8c8cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 29 22:56:17.512783 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:17.512753 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2hddb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:56:17.513258 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:17.512815 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" podUID="b1fb295f-b287-441f-a057-efaf16a8c8cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 29 22:56:18.512474 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:18.512440 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2hddb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:56:18.512625 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:18.512509 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" podUID="b1fb295f-b287-441f-a057-efaf16a8c8cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 29 22:56:19.512452 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.512406 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2hddb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:56:19.513050 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.513006 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" podUID="b1fb295f-b287-441f-a057-efaf16a8c8cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 29 22:56:19.675150 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.675051 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz"] Oct 29 22:56:19.680360 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.680331 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.740086 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.739112 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz"] Oct 29 22:56:19.744762 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.743999 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.744762 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.744046 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da46e2ae-5388-4445-8351-bcc2204a9dcf-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.744762 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.744120 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.744762 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.744186 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hx99\" (UniqueName: \"kubernetes.io/projected/da46e2ae-5388-4445-8351-bcc2204a9dcf-kube-api-access-2hx99\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.744762 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.744299 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.744762 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.744330 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.744762 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.744377 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.744762 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.744426 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.744762 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.744488 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.845775 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.845738 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.845775 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.845779 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.846023 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.845820 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.846023 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.845862 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.846023 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.845913 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.846190 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.846052 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.846190 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.846080 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da46e2ae-5388-4445-8351-bcc2204a9dcf-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.846190 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.846143 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.846350 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.846230 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hx99\" (UniqueName: \"kubernetes.io/projected/da46e2ae-5388-4445-8351-bcc2204a9dcf-kube-api-access-2hx99\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.846413 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.846347 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.846413 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.846369 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.846657 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.846614 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.846878 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.846847 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.847472 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.847439 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da46e2ae-5388-4445-8351-bcc2204a9dcf-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.848611 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.848588 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.848861 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.848834 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.869151 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.869119 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:19.870558 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:19.870538 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hx99\" (UniqueName: \"kubernetes.io/projected/da46e2ae-5388-4445-8351-bcc2204a9dcf-kube-api-access-2hx99\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:20.036922 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:20.036828 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:20.378235 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:20.378183 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz"] Oct 29 22:56:20.378381 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:56:20.378178 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda46e2ae_5388_4445_8351_bcc2204a9dcf.slice/crio-2679e738eda982de78374d8f8fe5e32c36d7a1903d537c8ebea66e14ba43709b WatchSource:0}: Error finding container 2679e738eda982de78374d8f8fe5e32c36d7a1903d537c8ebea66e14ba43709b: Status 404 returned error can't find the container with id 2679e738eda982de78374d8f8fe5e32c36d7a1903d537c8ebea66e14ba43709b Oct 29 22:56:20.380751 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:20.380713 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:56:20.380859 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:20.380786 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:56:20.380859 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:20.380817 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:56:20.458598 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:20.458577 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw" Oct 29 22:56:20.515223 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:20.514503 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-2hddb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:56:20.515223 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:20.514572 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" podUID="b1fb295f-b287-441f-a057-efaf16a8c8cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 29 22:56:20.564478 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:20.564425 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" event={"ID":"da46e2ae-5388-4445-8351-bcc2204a9dcf","Type":"ContainerStarted","Data":"4fc80bbdf569b4c193d1b0fa400f79242cf0a642dc0edaad6ef227e83a9061e0"} Oct 29 22:56:20.564638 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:20.564511 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" event={"ID":"da46e2ae-5388-4445-8351-bcc2204a9dcf","Type":"ContainerStarted","Data":"2679e738eda982de78374d8f8fe5e32c36d7a1903d537c8ebea66e14ba43709b"} Oct 29 22:56:20.612962 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:20.612660 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" podStartSLOduration=1.612640534 podStartE2EDuration="1.612640534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:20.608518214 +0000 UTC m=+834.549066124" watchObservedRunningTime="2026-10-29 22:56:20.612640534 +0000 UTC m=+834.553188445" Oct 29 22:56:21.043224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:21.042817 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:21.043224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:21.042950 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:56:21.043224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:21.042992 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" podUID="da46e2ae-5388-4445-8351-bcc2204a9dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 29 22:56:22.041517 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:22.037937 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:56:22.041517 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:22.037996 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" podUID="da46e2ae-5388-4445-8351-bcc2204a9dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 29 22:56:22.437225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:22.434539 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:22.437225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:22.436075 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:22.447481 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:22.447451 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 22:56:23.039679 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:23.039643 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:56:23.039868 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:23.039702 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" podUID="da46e2ae-5388-4445-8351-bcc2204a9dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 29 22:56:23.886397 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:56:23.886363 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:56:24.044368 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:24.044330 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:56:24.044658 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:24.044627 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" podUID="da46e2ae-5388-4445-8351-bcc2204a9dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 29 22:56:25.005961 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:25.005928 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5"] Oct 29 22:56:25.012017 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:25.011949 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5" Oct 29 22:56:25.035227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:25.033521 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a37a29-dockercfg-smvd7\"" Oct 29 22:56:25.035227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:25.033632 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a37a29\"" Oct 29 22:56:25.097779 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:25.087409 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:56:25.097779 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:25.087486 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" podUID="da46e2ae-5388-4445-8351-bcc2204a9dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 29 22:56:25.097779 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:25.097311 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5"] Oct 29 22:56:25.194781 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:25.194741 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xbbb\" (UniqueName: \"kubernetes.io/projected/eea051e6-e43a-473a-9c98-1a018a8e3d27-kube-api-access-7xbbb\") pod \"payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5\" (UID: \"eea051e6-e43a-473a-9c98-1a018a8e3d27\") " pod="openshift-ingress/payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5" Oct 29 22:56:25.194939 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:25.194802 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea051e6-e43a-473a-9c98-1a018a8e3d27-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5\" (UID: \"eea051e6-e43a-473a-9c98-1a018a8e3d27\") " pod="openshift-ingress/payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5" Oct 29 22:56:25.296753 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:25.296169 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xbbb\" (UniqueName: \"kubernetes.io/projected/eea051e6-e43a-473a-9c98-1a018a8e3d27-kube-api-access-7xbbb\") pod \"payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5\" (UID: \"eea051e6-e43a-473a-9c98-1a018a8e3d27\") " pod="openshift-ingress/payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5" Oct 29 22:56:25.296753 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:25.296239 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea051e6-e43a-473a-9c98-1a018a8e3d27-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5\" (UID: \"eea051e6-e43a-473a-9c98-1a018a8e3d27\") " pod="openshift-ingress/payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5" Oct 29 22:56:25.297299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:25.297272 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea051e6-e43a-473a-9c98-1a018a8e3d27-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5\" (UID: \"eea051e6-e43a-473a-9c98-1a018a8e3d27\") " pod="openshift-ingress/payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5" Oct 29 22:56:25.347775 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:25.347735 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xbbb\" (UniqueName: \"kubernetes.io/projected/eea051e6-e43a-473a-9c98-1a018a8e3d27-kube-api-access-7xbbb\") pod \"payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5\" (UID: \"eea051e6-e43a-473a-9c98-1a018a8e3d27\") " pod="openshift-ingress/payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5" Oct 29 22:56:25.494451 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:25.494422 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5" Oct 29 22:56:25.843756 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:25.843724 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5"] Oct 29 22:56:25.845778 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:56:25.845745 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea051e6_e43a_473a_9c98_1a018a8e3d27.slice/crio-ed64ec60128ac4332d45d6404323e249a2c979c575bd23e5f290f42941b05990 WatchSource:0}: Error finding container ed64ec60128ac4332d45d6404323e249a2c979c575bd23e5f290f42941b05990: Status 404 returned error can't find the container with id ed64ec60128ac4332d45d6404323e249a2c979c575bd23e5f290f42941b05990 Oct 29 22:56:26.039705 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:26.039658 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:56:26.040269 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:26.039727 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" podUID="da46e2ae-5388-4445-8351-bcc2204a9dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 29 22:56:26.684519 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:26.684475 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5" event={"ID":"eea051e6-e43a-473a-9c98-1a018a8e3d27","Type":"ContainerStarted","Data":"262493a48269796629dd147f0d447890f5ebfc1368d6944797b48c9983b6a28f"} Oct 29 22:56:26.684836 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:26.684569 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5" event={"ID":"eea051e6-e43a-473a-9c98-1a018a8e3d27","Type":"ContainerStarted","Data":"ed64ec60128ac4332d45d6404323e249a2c979c575bd23e5f290f42941b05990"} Oct 29 22:56:26.684836 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:26.684603 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5" Oct 29 22:56:26.727290 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:26.727236 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5" podStartSLOduration=2.727219779 podStartE2EDuration="2.727219779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:26.723289594 +0000 UTC m=+840.663837504" watchObservedRunningTime="2026-10-29 22:56:26.727219779 +0000 UTC m=+840.667767688" Oct 29 22:56:27.038169 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:27.038079 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 29 22:56:27.038169 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:27.038133 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" podUID="da46e2ae-5388-4445-8351-bcc2204a9dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 29 22:56:28.041896 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:28.041865 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:28.042318 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:28.042294 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:28.043129 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:28.043106 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 22:56:34.457505 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.457467 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd"] Oct 29 22:56:34.479494 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.479459 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:34.511611 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.511095 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f3092a4d\"/\"default-dockercfg-448zv\"" Oct 29 22:56:34.511611 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.511233 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f3092a4d\"/\"e2e-lifecycle-model-f3092a4d-kserve-self-signed-certs\"" Oct 29 22:56:34.517703 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.517668 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd"] Oct 29 22:56:34.540282 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.529271 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f3092a4d\"/\"kube-root-ca.crt\"" Oct 29 22:56:34.540282 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.529381 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f3092a4d\"/\"openshift-service-ca.crt\"" Oct 29 22:56:34.690275 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.690128 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq52z\" (UniqueName: \"kubernetes.io/projected/cb05c692-f4a5-49bf-a84d-ca3f03edd407-kube-api-access-bq52z\") pod \"e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd\" (UID: \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\") " pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:34.690477 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.690351 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-dshm\") pod \"e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd\" (UID: \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\") " pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:34.690477 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.690403 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb05c692-f4a5-49bf-a84d-ca3f03edd407-tls-certs\") pod \"e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd\" (UID: \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\") " pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:34.690614 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.690536 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-model-cache\") pod \"e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd\" (UID: \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\") " pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:34.690614 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.690579 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-tmp-dir\") pod \"e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd\" (UID: \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\") " pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:34.698237 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.690749 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-home\") pod \"e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd\" (UID: \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\") " pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:34.814584 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.814482 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-model-cache\") pod \"e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd\" (UID: \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\") " pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:34.814867 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.814824 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-tmp-dir\") pod \"e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd\" (UID: \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\") " pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:34.836224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.817818 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-home\") pod \"e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd\" (UID: \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\") " pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:34.836224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.815546 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-tmp-dir\") pod \"e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd\" (UID: \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\") " pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:34.836224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.815871 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-model-cache\") pod \"e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd\" (UID: \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\") " pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:34.836224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.818007 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq52z\" (UniqueName: \"kubernetes.io/projected/cb05c692-f4a5-49bf-a84d-ca3f03edd407-kube-api-access-bq52z\") pod \"e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd\" (UID: \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\") " pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:34.836224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.818145 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-dshm\") pod \"e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd\" (UID: \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\") " pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:34.836224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.818188 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb05c692-f4a5-49bf-a84d-ca3f03edd407-tls-certs\") pod \"e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd\" (UID: \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\") " pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:34.836224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.818351 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-home\") pod \"e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd\" (UID: \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\") " pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:34.863124 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.863091 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb05c692-f4a5-49bf-a84d-ca3f03edd407-tls-certs\") pod \"e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd\" (UID: \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\") " pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:34.863699 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.863675 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-dshm\") pod \"e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd\" (UID: \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\") " pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:34.866494 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:34.866470 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq52z\" (UniqueName: \"kubernetes.io/projected/cb05c692-f4a5-49bf-a84d-ca3f03edd407-kube-api-access-bq52z\") pod \"e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd\" (UID: \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\") " pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:35.027252 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:35.025704 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:35.825738 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:35.813060 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd"] Oct 29 22:56:35.985243 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:56:35.978655 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb05c692_f4a5_49bf_a84d_ca3f03edd407.slice/crio-9ff4be40bf66fcfc42ad77102bac67f0f887bbcbf8342614fbe7569f36a99ab0 WatchSource:0}: Error finding container 9ff4be40bf66fcfc42ad77102bac67f0f887bbcbf8342614fbe7569f36a99ab0: Status 404 returned error can't find the container with id 9ff4be40bf66fcfc42ad77102bac67f0f887bbcbf8342614fbe7569f36a99ab0 Oct 29 22:56:36.305224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.300590 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:36.334021 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.331321 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght"] Oct 29 22:56:36.346431 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.346402 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght"] Oct 29 22:56:36.346817 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.346795 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.517811 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.516551 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 29 22:56:36.570237 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.544618 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q5wl\" (UniqueName: \"kubernetes.io/projected/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-kube-api-access-5q5wl\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.570237 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.544678 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.570237 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.544705 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.570237 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.544734 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.570237 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.544869 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.570237 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.544932 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.570237 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.544971 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.570237 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.545050 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.570237 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.545128 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.608226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.601286 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:36.629425 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.629320 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 29 22:56:36.676733 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.645350 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:36.676733 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.638484 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:36.676733 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.660260 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.676733 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.660316 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.676733 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.660377 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.676733 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.660451 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.676733 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.660506 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q5wl\" (UniqueName: \"kubernetes.io/projected/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-kube-api-access-5q5wl\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.676733 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.660555 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.676733 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.660578 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.676733 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.660605 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.676733 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.660761 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.676733 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.669225 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.676733 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.669897 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.676733 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.670171 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.676733 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.670438 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.676733 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.670674 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.722226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.718776 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.833166 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.756823 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:36.833166 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.758020 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.857674 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.841809 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q5wl\" (UniqueName: \"kubernetes.io/projected/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-kube-api-access-5q5wl\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.873227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.870805 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:36.912737 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:36.912643 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:37.016889 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:37.016601 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:37.085275 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:37.076327 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:37.306225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:37.301696 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" event={"ID":"cb05c692-f4a5-49bf-a84d-ca3f03edd407","Type":"ContainerStarted","Data":"9ff4be40bf66fcfc42ad77102bac67f0f887bbcbf8342614fbe7569f36a99ab0"} Oct 29 22:56:37.306225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:37.305744 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 29 22:56:37.315836 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:37.310331 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:37.315836 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:37.310663 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:37.533245 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:37.521437 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:37.572266 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:37.559465 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:56:37.615238 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:37.613068 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:37.670987 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:37.658064 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:37.744785 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:37.717803 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:37.822227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:37.796066 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:37.863244 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:37.796313 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:56:37.863244 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:37.853270 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:56:37.863244 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:37.853490 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:56:37.961514 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:37.961463 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 29 22:56:38.023693 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:37.964449 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:56:38.188232 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.186415 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:38.333227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.330313 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:38.353085 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.342719 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:38.353085 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.352708 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:38.384927 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.384887 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:56:38.385326 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.385296 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" podUID="ae79f378-d055-4e08-bd3a-1e75fcc38882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:38.389914 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.389873 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:56:38.390162 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.390135 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 29 22:56:38.415232 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.411609 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sn4mm" podUID="bdeff50a-75a8-4272-9120-47c753cdcb5f" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.23:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:38.507235 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.468782 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:38.618390 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.618346 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:38.618999 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.618971 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 29 22:56:38.724228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.619271 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:38.724593 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.658531 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:38.795638 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.776904 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:56:38.795638 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.776996 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:56:38.795638 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.777038 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:56:38.795638 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.780072 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:38.880411 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.845313 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:38.940772 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.940734 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:56:38.974224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.949407 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" event={"ID":"cb05c692-f4a5-49bf-a84d-ca3f03edd407","Type":"ContainerStarted","Data":"6981b78dd69c1f162f07facba501c4fd64cd57ebbb19b4713df9502995776f48"} Oct 29 22:56:38.974513 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.974485 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:56:38.974702 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.974691 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:56:39.057954 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.057921 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:56:39.058218 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.942315 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:39.074226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.001488 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 29 22:56:39.074226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:38.976281 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:39.088336 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.088304 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5" Oct 29 22:56:39.101319 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.101288 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn"] Oct 29 22:56:39.163277 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.146945 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:39.163277 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.147008 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42"] Oct 29 22:56:39.163277 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.158386 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:39.236229 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.219004 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn"] Oct 29 22:56:39.614325 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.593758 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42"] Oct 29 22:56:39.614325 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.593802 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght"] Oct 29 22:56:39.614325 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.397428 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:39.614325 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.397457 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 29 22:56:39.614325 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.397484 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 29 22:56:39.614325 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.475285 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:39.614325 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.525810 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:56:39.614325 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.609294 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" podUID="30e8d47c-27fb-40d8-86fb-eea11610630b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:39.614325 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.226666 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:39.614325 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.396321 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:39.614325 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.610029 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:39.614325 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.610141 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6103a8bf-2a28-4991-8b55-fecb16515d7f-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:39.614325 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.610273 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:39.673549 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.618723 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:39.673549 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.618955 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:56:39.673549 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.647009 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:39.734101 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.715394 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:39.734101 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.715526 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdjr5\" (UniqueName: \"kubernetes.io/projected/6103a8bf-2a28-4991-8b55-fecb16515d7f-kube-api-access-xdjr5\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:39.734101 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.715591 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:39.734101 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.715615 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:39.734101 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.715656 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:39.734101 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.725254 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:39.734815 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.734348 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 29 22:56:39.940764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.837166 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:39.940764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.837283 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:39.940764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.837354 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:39.940764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.837447 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:39.940764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.837492 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdjr5\" (UniqueName: \"kubernetes.io/projected/6103a8bf-2a28-4991-8b55-fecb16515d7f-kube-api-access-xdjr5\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:39.940764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.837534 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:39.940764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.837586 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:39.940764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.837620 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62w66\" (UniqueName: \"kubernetes.io/projected/0bb6b077-1785-4673-9dc9-80258dcbecf3-kube-api-access-62w66\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:39.940764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.837651 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:39.940764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.837687 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:39.940764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.837709 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:39.940764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.837761 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6103a8bf-2a28-4991-8b55-fecb16515d7f-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:39.940764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.837787 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:39.940764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.837857 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:39.940764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.837955 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:39.940764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.837990 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:39.940764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.838021 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb6b077-1785-4673-9dc9-80258dcbecf3-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:39.940764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.838069 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:39.940764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.883168 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 29 22:56:39.940764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:39.903925 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:56:40.053800 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.053758 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Oct 29 22:56:40.138528 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.061102 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:40.404230 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.064952 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:40.404230 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.065657 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6103a8bf-2a28-4991-8b55-fecb16515d7f-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:40.404230 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.138810 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:40.404230 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.148784 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:40.404230 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.241282 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 29 22:56:40.418362 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.260351 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 29 22:56:40.420032 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.260391 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 29 22:56:40.436271 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.423826 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 22:56:40.436271 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.430580 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62w66\" (UniqueName: \"kubernetes.io/projected/0bb6b077-1785-4673-9dc9-80258dcbecf3-kube-api-access-62w66\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:40.436271 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.430633 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:40.436271 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.430671 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:40.436271 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.430779 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:40.436271 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.430886 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb6b077-1785-4673-9dc9-80258dcbecf3-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:40.436271 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.430933 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:40.436271 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.430991 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:40.436271 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.431053 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:40.436271 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.431237 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:40.436271 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.432315 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-workload-certs\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:40.504237 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.498337 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded" Oct 29 22:56:40.600253 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.566377 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-data\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:40.600253 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.598741 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 29 22:56:40.679600 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.615975 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:40.771225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.768477 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 29 22:56:40.771225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.768973 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 29 22:56:40.802626 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.801548 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-credential-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:40.802626 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.801734 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-workload-socket\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:40.802626 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.802221 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb6b077-1785-4673-9dc9-80258dcbecf3-istiod-ca-cert\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:40.803662 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.803635 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:40.861831 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.804119 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 29 22:56:40.862656 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.813282 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 29 22:56:40.863533 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:40.816450 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdjr5\" (UniqueName: \"kubernetes.io/projected/6103a8bf-2a28-4991-8b55-fecb16515d7f-kube-api-access-xdjr5\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:41.086228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.053746 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:41.141874 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.141836 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 29 22:56:41.192233 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.169836 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" event={"ID":"e02c68ae-98e8-47ce-8b21-9aa5025a20d5","Type":"ContainerStarted","Data":"bcae21405961c82c7b9efc0be67ba3054fcae2f30838ad2a071f965793a7144b"} Oct 29 22:56:41.260007 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.259913 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:56:41.260332 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.260312 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:56:41.260515 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.260500 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:56:41.298093 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.298059 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:41.335245 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.306605 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-envoy\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:41.339489 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.335992 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-podinfo\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:41.382523 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.382488 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62w66\" (UniqueName: \"kubernetes.io/projected/0bb6b077-1785-4673-9dc9-80258dcbecf3-kube-api-access-62w66\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:41.403870 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.403832 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:41.433229 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.405526 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:41.433587 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.433530 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:41.435296 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.435254 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 29 22:56:41.502318 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.482634 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded" Oct 29 22:56:41.565476 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.502602 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:41.566378 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.547642 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:41.566853 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.563688 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-token\") pod \"e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:41.574287 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.565302 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:41.668260 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.639754 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:41.701591 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.701454 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:41.796251 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.791926 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:41.796251 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.792315 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 29 22:56:41.894894 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.894822 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" podStartSLOduration=7.894795261 podStartE2EDuration="7.894795261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:39.116559804 +0000 UTC m=+853.057107728" watchObservedRunningTime="2026-10-29 22:56:41.894795261 +0000 UTC m=+855.835343172" Oct 29 22:56:41.930050 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:41.928278 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:42.075589 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:42.069106 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:42.148010 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:42.147723 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:42.200400 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:42.163525 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" event={"ID":"e02c68ae-98e8-47ce-8b21-9aa5025a20d5","Type":"ContainerStarted","Data":"e14e2622fb7b7c26fab34b5caa08a7d0d83b38b2e2728f1fbf0af8d90e7c0bd7"} Oct 29 22:56:42.248225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:42.165285 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:56:42.248225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:42.218764 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" podUID="7874a224-3fab-4fee-90db-89b753ed938f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:42.523179 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:42.489821 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:42.523179 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:42.490544 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 29 22:56:42.523179 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:42.491102 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:42.598373 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:42.598306 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 22:56:42.630829 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:42.629744 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 29 22:56:42.648496 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:42.643839 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:42.721351 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:42.704692 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:42.721351 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:42.706240 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:42.794579 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:42.792428 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:42.794579 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:42.792769 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 29 22:56:42.804426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:42.804366 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" podStartSLOduration=6.804347705 podStartE2EDuration="6.804347705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:42.360950027 +0000 UTC m=+856.301497938" watchObservedRunningTime="2026-10-29 22:56:42.804347705 +0000 UTC m=+856.744895619" Oct 29 22:56:42.888667 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:42.888627 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:56:42.889831 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:42.889805 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:56:42.890004 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:42.889984 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:56:42.891792 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:42.889494 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:56:42.891975 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:42.891957 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:56:42.892130 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:42.892115 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:56:42.892315 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:42.892300 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn"] Oct 29 22:56:43.048569 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:43.041801 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:43.048569 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:43.041921 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:56:43.048569 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:43.041966 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" podUID="e02c68ae-98e8-47ce-8b21-9aa5025a20d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 29 22:56:43.173738 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:43.172395 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:43.302873 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:43.302660 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:56:43.329848 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:43.329816 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af65e63-nndvh"] Oct 29 22:56:43.358097 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:43.358055 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" event={"ID":"6103a8bf-2a28-4991-8b55-fecb16515d7f","Type":"ContainerStarted","Data":"f8057c1c4dd5fea6a01404c2b672930a79da296c60055b5d7ff5b83ad0747a85"} Oct 29 22:56:43.361313 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:43.361226 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af65e63-nndvh" Oct 29 22:56:43.378984 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:43.378961 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:56:43.379916 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:43.379889 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 22:56:43.477909 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:43.477863 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07ef42d-e6d7-423c-8424-62efb7d851d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3af65e63-nndvh\" (UID: \"c07ef42d-e6d7-423c-8424-62efb7d851d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af65e63-nndvh" Oct 29 22:56:43.478183 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:43.478159 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h8tj\" (UniqueName: \"kubernetes.io/projected/c07ef42d-e6d7-423c-8424-62efb7d851d7-kube-api-access-4h8tj\") pod \"maas-api-revoke-sub-3af65e63-nndvh\" (UID: \"c07ef42d-e6d7-423c-8424-62efb7d851d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af65e63-nndvh" Oct 29 22:56:43.497669 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:43.497614 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 29 22:56:43.498046 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:43.497997 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:43.500882 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:43.500858 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 29 22:56:43.653688 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:43.653644 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:43.654425 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:43.654401 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:56:43.661591 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:43.660302 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:43.701623 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:43.675956 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:43.749227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:43.746870 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 29 22:56:43.749227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:43.747267 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 29 22:56:43.749227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:43.747794 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:56:43.903756 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:43.903665 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07ef42d-e6d7-423c-8424-62efb7d851d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3af65e63-nndvh\" (UID: \"c07ef42d-e6d7-423c-8424-62efb7d851d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af65e63-nndvh" Oct 29 22:56:44.041731 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:43.953862 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:56:44.042080 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.042060 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:56:44.042249 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.042233 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af65e63-nndvh"] Oct 29 22:56:44.067609 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.066162 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:56:44.067609 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.066368 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:56:44.067609 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.066474 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:56:44.068468 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.068432 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:56:44.068681 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.068653 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" podUID="e02c68ae-98e8-47ce-8b21-9aa5025a20d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 29 22:56:44.072118 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.072095 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:56:44.171187 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.171155 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:56:44.174028 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.174004 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:56:44.179754 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.179720 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07ef42d-e6d7-423c-8424-62efb7d851d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3af65e63-nndvh\" (UID: \"c07ef42d-e6d7-423c-8424-62efb7d851d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af65e63-nndvh" Oct 29 22:56:44.180015 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.179997 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07ef42d-e6d7-423c-8424-62efb7d851d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3af65e63-nndvh\" (UID: \"c07ef42d-e6d7-423c-8424-62efb7d851d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af65e63-nndvh" Oct 29 22:56:44.180143 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.180128 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h8tj\" (UniqueName: \"kubernetes.io/projected/c07ef42d-e6d7-423c-8424-62efb7d851d7-kube-api-access-4h8tj\") pod \"maas-api-revoke-sub-3af65e63-nndvh\" (UID: \"c07ef42d-e6d7-423c-8424-62efb7d851d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af65e63-nndvh" Oct 29 22:56:44.195264 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.185842 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07ef42d-e6d7-423c-8424-62efb7d851d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3af65e63-nndvh\" (UID: \"c07ef42d-e6d7-423c-8424-62efb7d851d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af65e63-nndvh" Oct 29 22:56:44.211425 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.211394 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h8tj\" (UniqueName: \"kubernetes.io/projected/c07ef42d-e6d7-423c-8424-62efb7d851d7-kube-api-access-4h8tj\") pod \"maas-api-revoke-sub-3af65e63-nndvh\" (UID: \"c07ef42d-e6d7-423c-8424-62efb7d851d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af65e63-nndvh" Oct 29 22:56:44.253228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.229368 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07ef42d-e6d7-423c-8424-62efb7d851d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3af65e63-nndvh\" (UID: \"c07ef42d-e6d7-423c-8424-62efb7d851d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af65e63-nndvh" Oct 29 22:56:44.254754 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.254247 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:56:44.255004 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.254990 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:56:44.255416 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.255401 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:56:44.273770 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.273654 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42"] Oct 29 22:56:44.279787 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.279749 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv"] Oct 29 22:56:44.286395 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.286372 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.290245 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.290188 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv"] Oct 29 22:56:44.303910 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.303886 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f3092a4d-openshift-default-dockercfg-b9jst\"" Oct 29 22:56:44.390054 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.383480 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-data\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.390054 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.383709 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-token\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.390054 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.383869 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-workload-certs\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.390054 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.383898 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istiod-ca-cert\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.390054 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.383927 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-podinfo\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.390054 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.383989 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-workload-socket\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.390054 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.384010 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-envoy\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.390054 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.384031 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-credential-socket\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.390054 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.384145 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8nln\" (UniqueName: \"kubernetes.io/projected/74ab0e3b-a727-4b4f-82b4-56712d0f448a-kube-api-access-v8nln\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.401469 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.401434 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" event={"ID":"6103a8bf-2a28-4991-8b55-fecb16515d7f","Type":"ContainerStarted","Data":"23cf050765136816912dd67848761598c1b3143d1d17aa6109ba0e68ae56ec19"} Oct 29 22:56:44.404014 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.403944 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" event={"ID":"0bb6b077-1785-4673-9dc9-80258dcbecf3","Type":"ContainerStarted","Data":"30cbd3ed625e2a0b62416d2e3dc36d8e0017c97b07d1761b17316181534660d4"} Oct 29 22:56:44.454001 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.452631 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af65e63-nndvh" Oct 29 22:56:44.499226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.489552 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-token\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.499226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.489733 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-workload-certs\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.499226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.489778 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istiod-ca-cert\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.499226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.489824 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-podinfo\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.499226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.489872 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-workload-socket\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.499226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.489896 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-envoy\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.499226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.489921 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-credential-socket\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.499226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.490031 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8nln\" (UniqueName: \"kubernetes.io/projected/74ab0e3b-a727-4b4f-82b4-56712d0f448a-kube-api-access-v8nln\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.499226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.490069 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-data\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.499226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.491061 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-workload-socket\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.499226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.493459 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-workload-certs\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.499226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.493718 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-data\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.499226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.493951 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-credential-socket\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.499226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.494048 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-envoy\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.499226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.495522 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istiod-ca-cert\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.499226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.496439 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-podinfo\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.559235 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.557437 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:44.587082 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.587024 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-token\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.590100 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.589956 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8nln\" (UniqueName: \"kubernetes.io/projected/74ab0e3b-a727-4b4f-82b4-56712d0f448a-kube-api-access-v8nln\") pod \"e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.643139 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.643076 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" podStartSLOduration=6.643056165 podStartE2EDuration="6.643056165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:44.635556738 +0000 UTC m=+858.576104649" watchObservedRunningTime="2026-10-29 22:56:44.643056165 +0000 UTC m=+858.583604076" Oct 29 22:56:44.708779 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.706969 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:44.962064 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:44.962002 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af65e63-nndvh"] Oct 29 22:56:45.037224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:45.029553 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:45.047996 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:45.047551 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:56:45.047996 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:45.047612 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" podUID="e02c68ae-98e8-47ce-8b21-9aa5025a20d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 29 22:56:45.062331 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:45.062301 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:45.252479 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:45.252311 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv"] Oct 29 22:56:45.256461 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:45.256422 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:56:45.256573 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:45.256515 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:56:45.256608 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:45.256565 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:56:45.479140 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:45.478544 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" event={"ID":"0bb6b077-1785-4673-9dc9-80258dcbecf3","Type":"ContainerStarted","Data":"c56926b65dfa34cafd80fac4c455899fe4bf67e7a20f0cc082eb47b0e159fa6d"} Oct 29 22:56:45.567284 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:45.563629 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af65e63-nndvh" event={"ID":"c07ef42d-e6d7-423c-8424-62efb7d851d7","Type":"ContainerStarted","Data":"2c62bcd60a13bf986c6f8325e18a619a2cad7337921c410b4c4d888ad82cd8a3"} Oct 29 22:56:45.567284 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:45.563706 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af65e63-nndvh" event={"ID":"c07ef42d-e6d7-423c-8424-62efb7d851d7","Type":"ContainerStarted","Data":"df669881e2905b37c012c8968970d86e550b6533a2eb8933398037ca26fed153"} Oct 29 22:56:45.590060 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:45.589741 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:56:45.590060 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:45.589799 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" podUID="6103a8bf-2a28-4991-8b55-fecb16515d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 29 22:56:45.597232 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:45.596039 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" event={"ID":"74ab0e3b-a727-4b4f-82b4-56712d0f448a","Type":"ContainerStarted","Data":"04fad4b405b2c3fd5dea1b9cd65bf39990b36cc9bb018d347c8ef7f72b2bb733"} Oct 29 22:56:45.597232 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:45.596096 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" event={"ID":"74ab0e3b-a727-4b4f-82b4-56712d0f448a","Type":"ContainerStarted","Data":"c7c41e1281e0f20253291bc5e312d9f33fc552f603b6f54292e53d6e00bc7962"} Oct 29 22:56:45.699225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:45.698597 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" podStartSLOduration=7.698574069 podStartE2EDuration="7.698574069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:45.564985339 +0000 UTC m=+859.505533251" watchObservedRunningTime="2026-10-29 22:56:45.698574069 +0000 UTC m=+859.639121982" Oct 29 22:56:45.699448 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:45.699232 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af65e63-nndvh" podStartSLOduration=2.699216993 podStartE2EDuration="2.699216993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:45.609275065 +0000 UTC m=+859.549822976" watchObservedRunningTime="2026-10-29 22:56:45.699216993 +0000 UTC m=+859.639764898" Oct 29 22:56:45.710566 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:45.709958 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:45.748340 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:45.746510 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" podStartSLOduration=2.74648822 podStartE2EDuration="2.74648822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:45.666678157 +0000 UTC m=+859.607226067" watchObservedRunningTime="2026-10-29 22:56:45.74648822 +0000 UTC m=+859.687036131" Oct 29 22:56:46.067225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:46.065924 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:56:46.067225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:46.066009 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" podUID="e02c68ae-98e8-47ce-8b21-9aa5025a20d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 29 22:56:46.104149 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:46.088075 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:46.163437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:46.150403 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:56:46.163437 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:46.150474 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" podUID="0bb6b077-1785-4673-9dc9-80258dcbecf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 29 22:56:46.361282 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:46.352874 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:46.553231 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:46.553177 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:56:46.604351 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:46.604303 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" podUID="6103a8bf-2a28-4991-8b55-fecb16515d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 29 22:56:46.714100 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:46.713930 4821 patch_prober.go:29] interesting pod/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:56:46.721615 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:46.721574 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" podUID="74ab0e3b-a727-4b4f-82b4-56712d0f448a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 29 22:56:46.765717 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:46.764752 4821 generic.go:356] "Generic (PLEG): container finished" podID="c07ef42d-e6d7-423c-8424-62efb7d851d7" containerID="2c62bcd60a13bf986c6f8325e18a619a2cad7337921c410b4c4d888ad82cd8a3" exitCode=0 Oct 29 22:56:46.766109 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:46.765959 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af65e63-nndvh" event={"ID":"c07ef42d-e6d7-423c-8424-62efb7d851d7","Type":"ContainerDied","Data":"2c62bcd60a13bf986c6f8325e18a619a2cad7337921c410b4c4d888ad82cd8a3"} Oct 29 22:56:47.022294 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:47.020345 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:56:47.022294 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:47.020422 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" podUID="e02c68ae-98e8-47ce-8b21-9aa5025a20d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 29 22:56:47.094903 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:47.094324 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:56:47.094903 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:47.094398 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" podUID="0bb6b077-1785-4673-9dc9-80258dcbecf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 29 22:56:47.550235 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:47.549596 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:56:47.550235 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:47.549659 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" podUID="6103a8bf-2a28-4991-8b55-fecb16515d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 29 22:56:47.602227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:47.601285 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" podUID="cb05c692-f4a5-49bf-a84d-ca3f03edd407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 29 22:56:47.627228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:47.610352 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:47.654228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:47.653533 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:47.712227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:47.711592 4821 patch_prober.go:29] interesting pod/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:56:47.712227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:47.711656 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" podUID="74ab0e3b-a727-4b4f-82b4-56712d0f448a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 29 22:56:47.755226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:47.754573 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:47.756234 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:47.755425 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:47.885228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:47.884331 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:48.049228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.005420 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 29 22:56:48.049228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.028573 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:56:48.049228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.028629 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" podUID="e02c68ae-98e8-47ce-8b21-9aa5025a20d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 29 22:56:48.152283 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.079831 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:56:48.152283 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.079887 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" podUID="0bb6b077-1785-4673-9dc9-80258dcbecf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 29 22:56:48.152283 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.101840 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:48.152283 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.124278 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 29 22:56:48.152283 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.125318 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 29 22:56:48.152926 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.152847 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:48.196984 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.193366 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:48.413072 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.411308 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:48.552940 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.552891 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:56:48.553540 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.553496 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" podUID="6103a8bf-2a28-4991-8b55-fecb16515d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 29 22:56:48.595564 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.595524 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:48.698282 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.695460 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:48.698282 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.697145 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:56:48.698282 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.698138 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:48.715240 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.708503 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" podUID="cb05c692-f4a5-49bf-a84d-ca3f03edd407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 29 22:56:48.739408 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.739375 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:56:48.739709 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.739693 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:48.741683 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.741654 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:48.758226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.757410 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:48.758746 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.758697 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 29 22:56:48.759443 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.759420 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:56:48.759720 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.759707 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:56:48.829319 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.827988 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:48.829319 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.808016 4821 patch_prober.go:29] interesting pod/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:56:48.829319 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:48.828064 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" podUID="74ab0e3b-a727-4b4f-82b4-56712d0f448a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 29 22:56:49.015396 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.015295 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:49.043694 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.043661 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:56:49.090599 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.090492 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 22:56:49.112624 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.091694 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:56:49.112624 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.097944 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:56:49.112624 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.097989 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" podUID="e02c68ae-98e8-47ce-8b21-9aa5025a20d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 29 22:56:49.112624 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.103420 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:56:49.112624 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.103461 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" podUID="0bb6b077-1785-4673-9dc9-80258dcbecf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 29 22:56:49.248660 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.241381 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:49.329487 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.268310 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:49.329487 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.268352 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:49.329487 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.268403 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 29 22:56:49.329487 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.246314 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 29 22:56:49.330880 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.304569 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:49.410253 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.398124 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:49.467952 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.467909 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:56:49.525259 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.515633 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:56:49.525259 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.515714 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:56:49.525259 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.515775 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:56:49.525259 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.515832 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:56:49.525259 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.515887 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:56:49.525259 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.489144 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:49.661226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.657656 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:56:49.705226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.699394 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:49.778359 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.778301 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:49.819228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.815420 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" podUID="cb05c692-f4a5-49bf-a84d-ca3f03edd407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:49.946937 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.946840 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:49.949238 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:49.819932 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:56:50.017711 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.017663 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:50.018393 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.018353 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:50.036840 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.036793 4821 patch_prober.go:29] interesting pod/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:56:50.037125 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.037084 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" podUID="74ab0e3b-a727-4b4f-82b4-56712d0f448a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 29 22:56:50.060263 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.044748 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af65e63-nndvh" Oct 29 22:56:50.060263 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.048954 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:56:50.060263 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.049005 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" podUID="6103a8bf-2a28-4991-8b55-fecb16515d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 29 22:56:50.060263 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.049612 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 29 22:56:50.060263 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.049659 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" podUID="e02c68ae-98e8-47ce-8b21-9aa5025a20d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 29 22:56:50.170262 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.162418 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:56:50.170262 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.162881 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Oct 29 22:56:50.171219 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.171162 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:50.212313 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.171964 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:56:50.212620 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.212582 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" podUID="0bb6b077-1785-4673-9dc9-80258dcbecf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 29 22:56:50.364245 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.348260 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c07ef42d-e6d7-423c-8424-62efb7d851d7-kube-api-access-4h8tj\" (UniqueName: \"kubernetes.io/projected/c07ef42d-e6d7-423c-8424-62efb7d851d7-kube-api-access-4h8tj\") pod \"c07ef42d-e6d7-423c-8424-62efb7d851d7\" (UID: \"c07ef42d-e6d7-423c-8424-62efb7d851d7\") " Oct 29 22:56:50.441434 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.441398 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c07ef42d-e6d7-423c-8424-62efb7d851d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07ef42d-e6d7-423c-8424-62efb7d851d7-maas-api-openshift-ca\") pod \"c07ef42d-e6d7-423c-8424-62efb7d851d7\" (UID: \"c07ef42d-e6d7-423c-8424-62efb7d851d7\") " Oct 29 22:56:50.441718 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.441697 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c07ef42d-e6d7-423c-8424-62efb7d851d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07ef42d-e6d7-423c-8424-62efb7d851d7-maas-api-xks-ca\") pod \"c07ef42d-e6d7-423c-8424-62efb7d851d7\" (UID: \"c07ef42d-e6d7-423c-8424-62efb7d851d7\") " Oct 29 22:56:50.443173 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.383866 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 29 22:56:50.444707 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.426324 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:50.461227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.427627 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:50.471470 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.428311 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:50.472170 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.443996 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c07ef42d-e6d7-423c-8424-62efb7d851d7-maas-api-openshift-ca" pod "c07ef42d-e6d7-423c-8424-62efb7d851d7" (UID: "c07ef42d-e6d7-423c-8424-62efb7d851d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:56:50.472340 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.383495 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 29 22:56:50.498497 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.383834 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 29 22:56:50.519067 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.519023 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:50.704403 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.704358 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:50.826281 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.816373 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:50.916228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.897558 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07ef42d-e6d7-423c-8424-62efb7d851d7-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:56:50.933352 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.933304 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" podUID="cb05c692-f4a5-49bf-a84d-ca3f03edd407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:50.995749 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:50.995702 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:51.070346 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.017892 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded" Oct 29 22:56:51.070346 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.023476 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:51.070346 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.025796 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:51.081311 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.081224 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c07ef42d-e6d7-423c-8424-62efb7d851d7-maas-api-xks-ca" pod "c07ef42d-e6d7-423c-8424-62efb7d851d7" (UID: "c07ef42d-e6d7-423c-8424-62efb7d851d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:56:51.110227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.110094 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07ef42d-e6d7-423c-8424-62efb7d851d7-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:56:51.192411 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.192369 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 29 22:56:51.343283 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.310954 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:56:51.636595 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.636556 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" podUID="0bb6b077-1785-4673-9dc9-80258dcbecf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 29 22:56:51.636896 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.358010 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 29 22:56:51.637346 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:56:51.457095 4821 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.294s" Oct 29 22:56:51.669883 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.669859 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:56:51.677551 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.484394 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:56:51.681347 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.677505 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af65e63-nndvh" event={"ID":"c07ef42d-e6d7-423c-8424-62efb7d851d7","Type":"ContainerDied","Data":"df669881e2905b37c012c8968970d86e550b6533a2eb8933398037ca26fed153"} Oct 29 22:56:51.681580 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.681552 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df669881e2905b37c012c8968970d86e550b6533a2eb8933398037ca26fed153" Oct 29 22:56:51.710256 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.528333 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 29 22:56:51.710256 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.357965 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 22:56:51.710256 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.376400 4821 patch_prober.go:29] interesting pod/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:56:51.710256 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.705895 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" podUID="74ab0e3b-a727-4b4f-82b4-56712d0f448a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 29 22:56:51.710256 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.461544 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 29 22:56:51.710256 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.669627 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af65e63-nndvh" Oct 29 22:56:51.710256 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.527322 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 29 22:56:51.710256 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.390420 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:56:51.710256 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.706649 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" podUID="6103a8bf-2a28-4991-8b55-fecb16515d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 29 22:56:51.710256 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.593573 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07ef42d-e6d7-423c-8424-62efb7d851d7-kube-api-access-4h8tj" pod "c07ef42d-e6d7-423c-8424-62efb7d851d7" (UID: "c07ef42d-e6d7-423c-8424-62efb7d851d7"). InnerVolumeSpecName "kube-api-access-4h8tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:56:51.710256 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.706906 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:51.710256 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.527286 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 29 22:56:51.711480 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.711462 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:51.711971 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.711955 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:56:51.712333 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.712307 4821 patch_prober.go:29] interesting pod/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:56:51.712428 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.712350 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" podUID="74ab0e3b-a727-4b4f-82b4-56712d0f448a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 29 22:56:51.712503 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.712423 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:56:51.712503 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.712447 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" podUID="6103a8bf-2a28-4991-8b55-fecb16515d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 29 22:56:51.762180 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.762153 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:51.796348 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.762745 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 22:56:51.799672 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.799649 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:51.800402 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.800379 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 22:56:51.901282 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.892034 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:56:51.939256 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.935590 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" podUID="cb05c692-f4a5-49bf-a84d-ca3f03edd407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 29 22:56:51.996364 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:51.996335 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:56:52.006287 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:52.002636 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:56:52.006287 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:52.002690 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:56:52.008887 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:52.008862 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:56:52.086228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:52.080750 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h8tj\" (UniqueName: \"kubernetes.io/projected/c07ef42d-e6d7-423c-8424-62efb7d851d7-kube-api-access-4h8tj\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:56:52.092740 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:52.092708 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 29 22:56:52.092953 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:52.092929 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" podUID="0bb6b077-1785-4673-9dc9-80258dcbecf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 29 22:56:52.212077 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:52.211975 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:56:52.288715 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:52.288688 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:56:52.290495 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:52.290473 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:56:52.291333 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:52.291318 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:56:52.292121 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:52.292104 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:56:52.370413 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:52.370382 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:56:52.389016 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:52.388982 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:56:52.568688 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:52.568584 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 29 22:56:52.568688 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:52.568649 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" podUID="6103a8bf-2a28-4991-8b55-fecb16515d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 29 22:56:52.750228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:52.737148 4821 patch_prober.go:29] interesting pod/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 29 22:56:52.750228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:52.737227 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" podUID="74ab0e3b-a727-4b4f-82b4-56712d0f448a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 29 22:56:53.526783 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.526740 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct"] Oct 29 22:56:53.527801 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.527780 4821 cpu_manager.go:414] "containerMap: removing container" podUID="c07ef42d-e6d7-423c-8424-62efb7d851d7" containerName="revoke-keys" Oct 29 22:56:53.527929 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.527917 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c07ef42d-e6d7-423c-8424-62efb7d851d7" containerName="revoke-keys" Oct 29 22:56:53.528232 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.528218 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="c07ef42d-e6d7-423c-8424-62efb7d851d7" containerName="revoke-keys" Oct 29 22:56:53.562091 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.551395 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct" Oct 29 22:56:53.562091 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.558957 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct"] Oct 29 22:56:53.614038 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.614007 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a37a29-serving-cert\"" Oct 29 22:56:53.717380 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.714797 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/128877c1-c77d-4d28-abc6-f49c24661c4a-openshift-service-ca\") pod \"maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct\" (UID: \"128877c1-c77d-4d28-abc6-f49c24661c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct" Oct 29 22:56:53.717380 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.717107 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/128877c1-c77d-4d28-abc6-f49c24661c4a-maas-api-tls\") pod \"maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct\" (UID: \"128877c1-c77d-4d28-abc6-f49c24661c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct" Oct 29 22:56:53.717380 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.717169 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/128877c1-c77d-4d28-abc6-f49c24661c4a-metrics-certs\") pod \"maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct\" (UID: \"128877c1-c77d-4d28-abc6-f49c24661c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct" Oct 29 22:56:53.717380 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.717194 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjrz7\" (UniqueName: \"kubernetes.io/projected/128877c1-c77d-4d28-abc6-f49c24661c4a-kube-api-access-jjrz7\") pod \"maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct\" (UID: \"128877c1-c77d-4d28-abc6-f49c24661c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct" Oct 29 22:56:53.765816 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.765769 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:53.768069 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.768012 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:53.769596 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.769577 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 22:56:53.819249 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.819148 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/128877c1-c77d-4d28-abc6-f49c24661c4a-maas-api-tls\") pod \"maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct\" (UID: \"128877c1-c77d-4d28-abc6-f49c24661c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct" Oct 29 22:56:53.819410 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.819323 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/128877c1-c77d-4d28-abc6-f49c24661c4a-metrics-certs\") pod \"maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct\" (UID: \"128877c1-c77d-4d28-abc6-f49c24661c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct" Oct 29 22:56:53.819410 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.819356 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjrz7\" (UniqueName: \"kubernetes.io/projected/128877c1-c77d-4d28-abc6-f49c24661c4a-kube-api-access-jjrz7\") pod \"maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct\" (UID: \"128877c1-c77d-4d28-abc6-f49c24661c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct" Oct 29 22:56:53.819549 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.819455 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/128877c1-c77d-4d28-abc6-f49c24661c4a-openshift-service-ca\") pod \"maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct\" (UID: \"128877c1-c77d-4d28-abc6-f49c24661c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct" Oct 29 22:56:53.820418 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.820390 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/128877c1-c77d-4d28-abc6-f49c24661c4a-openshift-service-ca\") pod \"maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct\" (UID: \"128877c1-c77d-4d28-abc6-f49c24661c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct" Oct 29 22:56:53.828324 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.827855 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/128877c1-c77d-4d28-abc6-f49c24661c4a-maas-api-tls\") pod \"maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct\" (UID: \"128877c1-c77d-4d28-abc6-f49c24661c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct" Oct 29 22:56:53.828324 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.828291 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/128877c1-c77d-4d28-abc6-f49c24661c4a-metrics-certs\") pod \"maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct\" (UID: \"128877c1-c77d-4d28-abc6-f49c24661c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct" Oct 29 22:56:53.846045 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.845933 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjrz7\" (UniqueName: \"kubernetes.io/projected/128877c1-c77d-4d28-abc6-f49c24661c4a-kube-api-access-jjrz7\") pod \"maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct\" (UID: \"128877c1-c77d-4d28-abc6-f49c24661c4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct" Oct 29 22:56:53.919012 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:56:53.918923 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:56:53.930690 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.929700 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct" Oct 29 22:56:53.965335 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.965311 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:53.968247 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.968080 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:53.978247 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.978224 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 22:56:53.988513 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.988489 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:53.988818 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.988798 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:53.989747 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:53.989729 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 22:56:54.334741 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:54.333384 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct"] Oct 29 22:56:54.340186 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:54.340060 4821 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:56:54.715801 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:54.715764 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct" event={"ID":"128877c1-c77d-4d28-abc6-f49c24661c4a","Type":"ContainerStarted","Data":"a08faf52c1c010b2b27fd6804acd1dac76adbfaa0c5106b2807cd2780356ef33"} Oct 29 22:56:55.815464 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:55.811598 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct" event={"ID":"128877c1-c77d-4d28-abc6-f49c24661c4a","Type":"ContainerStarted","Data":"d6d481d30083835ef6e4a575fd5379c0b5fcb927882303ed6293f6891dbe6cd4"} Oct 29 22:56:55.815464 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:55.812877 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct" Oct 29 22:56:55.895888 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:55.888497 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct" podStartSLOduration=2.470578389 podStartE2EDuration="2.888474889s" totalImagesPullingTime="417.8965ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:55.881790518 +0000 UTC m=+869.822338433" watchObservedRunningTime="2026-10-29 22:56:55.888474889 +0000 UTC m=+869.829022801" Oct 29 22:56:56.865887 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:56.865856 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b"] Oct 29 22:56:56.870547 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:56.870521 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:56.888311 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:56.888286 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e4a049-openshift-default-dockercfg-msz4p\"" Oct 29 22:56:56.905343 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:56.905315 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b"] Oct 29 22:56:56.911106 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:56.911069 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-podinfo\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:56.911324 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:56.911187 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkdk7\" (UniqueName: \"kubernetes.io/projected/c49af22e-9041-4f66-a3ff-175ad6d7b342-kube-api-access-tkdk7\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:56.911324 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:56.911271 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-workload-certs\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:56.911467 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:56.911397 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-token\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:56.911537 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:56.911481 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-data\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:56.911592 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:56.911556 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-credential-socket\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:56.911641 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:56.911601 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49af22e-9041-4f66-a3ff-175ad6d7b342-istiod-ca-cert\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:56.911692 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:56.911637 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-envoy\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:56.912001 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:56.911827 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-workload-socket\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:57.013415 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.013379 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-token\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:57.013620 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.013450 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-data\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:57.013620 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.013527 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-credential-socket\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:57.013620 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.013567 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49af22e-9041-4f66-a3ff-175ad6d7b342-istiod-ca-cert\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:57.013620 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.013606 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-envoy\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:57.013853 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.013720 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-workload-socket\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:57.013853 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.013784 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-podinfo\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:57.013853 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.013830 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkdk7\" (UniqueName: \"kubernetes.io/projected/c49af22e-9041-4f66-a3ff-175ad6d7b342-kube-api-access-tkdk7\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:57.013993 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.013878 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-workload-certs\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:57.013993 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.013937 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-data\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:57.013993 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.013970 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-credential-socket\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:57.014262 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.014178 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-workload-certs\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:57.014416 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.014387 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49af22e-9041-4f66-a3ff-175ad6d7b342-istiod-ca-cert\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:57.014578 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.014556 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-workload-socket\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:57.016542 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.016517 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-envoy\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:57.016706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.016686 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-podinfo\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:57.036049 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.036019 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-token\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:57.036320 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.036298 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkdk7\" (UniqueName: \"kubernetes.io/projected/c49af22e-9041-4f66-a3ff-175ad6d7b342-kube-api-access-tkdk7\") pod \"e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:57.186460 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.186373 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:57.534154 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.534114 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:56:57.534289 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.534187 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:56:57.534289 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.534259 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:56:57.536489 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.536462 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b"] Oct 29 22:56:57.881228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.879323 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" event={"ID":"c49af22e-9041-4f66-a3ff-175ad6d7b342","Type":"ContainerStarted","Data":"d2a8b524ca4f295ef4c469ea4eaf324974d9e43c62f9eaa134b81e92c2aa8a2c"} Oct 29 22:56:57.881228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.879427 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" event={"ID":"c49af22e-9041-4f66-a3ff-175ad6d7b342","Type":"ContainerStarted","Data":"eb6b13c88aa1179fcd77043479cef279cc3a0470e57a2ff9cd1d2eed59b22791"} Oct 29 22:56:57.963428 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:57.963368 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" podStartSLOduration=1.9633531020000001 podStartE2EDuration="1.963353102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:57.947357638 +0000 UTC m=+871.887905548" watchObservedRunningTime="2026-10-29 22:56:57.963353102 +0000 UTC m=+871.903901011" Oct 29 22:56:58.187259 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:58.187152 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:56:58.188740 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:58.188715 4821 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:56:58.188894 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:58.188765 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" podUID="c49af22e-9041-4f66-a3ff-175ad6d7b342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 29 22:56:59.186863 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:59.186825 4821 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:56:59.187368 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:56:59.186888 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" podUID="c49af22e-9041-4f66-a3ff-175ad6d7b342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 29 22:57:00.186806 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:00.186769 4821 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:57:00.187020 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:00.186830 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" podUID="c49af22e-9041-4f66-a3ff-175ad6d7b342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 29 22:57:01.186698 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:01.186661 4821 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:57:01.186905 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:01.186721 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" podUID="c49af22e-9041-4f66-a3ff-175ad6d7b342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 29 22:57:02.186743 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:02.186708 4821 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:57:02.187129 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:02.186782 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" podUID="c49af22e-9041-4f66-a3ff-175ad6d7b342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 29 22:57:02.885324 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:02.885289 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct" Oct 29 22:57:03.187354 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:03.187261 4821 patch_prober.go:29] interesting pod/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 29 22:57:03.187354 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:03.187312 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" podUID="c49af22e-9041-4f66-a3ff-175ad6d7b342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 29 22:57:04.191647 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:04.191610 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:57:04.192132 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:04.191998 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:57:04.192738 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:04.192716 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:57:06.676721 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:06.676682 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86"] Oct 29 22:57:06.681390 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:06.681370 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86" Oct 29 22:57:06.705642 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:06.705622 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-27d16a\"" Oct 29 22:57:06.711479 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:06.711454 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-27d16a-dockercfg-4jtlk\"" Oct 29 22:57:06.728260 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:06.728232 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86"] Oct 29 22:57:06.773508 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:06.773474 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72"] Oct 29 22:57:06.779619 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:06.779592 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72" Oct 29 22:57:06.843227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:06.833887 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krg76\" (UniqueName: \"kubernetes.io/projected/8c859b9a-7e01-4662-8fde-84242e0c387d-kube-api-access-krg76\") pod \"payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86\" (UID: \"8c859b9a-7e01-4662-8fde-84242e0c387d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86" Oct 29 22:57:06.843227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:06.840512 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c859b9a-7e01-4662-8fde-84242e0c387d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86\" (UID: \"8c859b9a-7e01-4662-8fde-84242e0c387d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86" Oct 29 22:57:06.888283 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:06.888251 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72"] Oct 29 22:57:06.943771 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:06.943663 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hksw6\" (UniqueName: \"kubernetes.io/projected/478131d3-996c-4d45-ab02-52cf354b8de9-kube-api-access-hksw6\") pod \"payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72\" (UID: \"478131d3-996c-4d45-ab02-52cf354b8de9\") " pod="openshift-ingress/payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72" Oct 29 22:57:06.943948 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:06.943812 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krg76\" (UniqueName: \"kubernetes.io/projected/8c859b9a-7e01-4662-8fde-84242e0c387d-kube-api-access-krg76\") pod \"payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86\" (UID: \"8c859b9a-7e01-4662-8fde-84242e0c387d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86" Oct 29 22:57:06.944097 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:06.944076 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c859b9a-7e01-4662-8fde-84242e0c387d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86\" (UID: \"8c859b9a-7e01-4662-8fde-84242e0c387d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86" Oct 29 22:57:06.944310 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:06.944281 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478131d3-996c-4d45-ab02-52cf354b8de9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72\" (UID: \"478131d3-996c-4d45-ab02-52cf354b8de9\") " pod="openshift-ingress/payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72" Oct 29 22:57:06.959942 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:06.959902 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c859b9a-7e01-4662-8fde-84242e0c387d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86\" (UID: \"8c859b9a-7e01-4662-8fde-84242e0c387d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86" Oct 29 22:57:06.999932 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:06.999898 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krg76\" (UniqueName: \"kubernetes.io/projected/8c859b9a-7e01-4662-8fde-84242e0c387d-kube-api-access-krg76\") pod \"payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86\" (UID: \"8c859b9a-7e01-4662-8fde-84242e0c387d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86" Oct 29 22:57:07.045539 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:07.045507 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478131d3-996c-4d45-ab02-52cf354b8de9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72\" (UID: \"478131d3-996c-4d45-ab02-52cf354b8de9\") " pod="openshift-ingress/payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72" Oct 29 22:57:07.045745 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:07.045569 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hksw6\" (UniqueName: \"kubernetes.io/projected/478131d3-996c-4d45-ab02-52cf354b8de9-kube-api-access-hksw6\") pod \"payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72\" (UID: \"478131d3-996c-4d45-ab02-52cf354b8de9\") " pod="openshift-ingress/payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72" Oct 29 22:57:07.046458 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:07.046428 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478131d3-996c-4d45-ab02-52cf354b8de9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72\" (UID: \"478131d3-996c-4d45-ab02-52cf354b8de9\") " pod="openshift-ingress/payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72" Oct 29 22:57:07.069804 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:07.069775 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hksw6\" (UniqueName: \"kubernetes.io/projected/478131d3-996c-4d45-ab02-52cf354b8de9-kube-api-access-hksw6\") pod \"payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72\" (UID: \"478131d3-996c-4d45-ab02-52cf354b8de9\") " pod="openshift-ingress/payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72" Oct 29 22:57:07.197810 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:07.197715 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72" Oct 29 22:57:07.296350 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:07.296304 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86" Oct 29 22:57:07.519408 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:07.519379 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72"] Oct 29 22:57:07.519689 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:57:07.519664 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod478131d3_996c_4d45_ab02_52cf354b8de9.slice/crio-ee04797d476f7091093d89217ef16408daa8d6c7dcc507dc69426514d3cff173 WatchSource:0}: Error finding container ee04797d476f7091093d89217ef16408daa8d6c7dcc507dc69426514d3cff173: Status 404 returned error can't find the container with id ee04797d476f7091093d89217ef16408daa8d6c7dcc507dc69426514d3cff173 Oct 29 22:57:07.634896 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:57:07.634867 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c859b9a_7e01_4662_8fde_84242e0c387d.slice/crio-5a37868f2acadc031c6932af5c4e34ce77331aafc54d66c32b73455cbea08ef0 WatchSource:0}: Error finding container 5a37868f2acadc031c6932af5c4e34ce77331aafc54d66c32b73455cbea08ef0: Status 404 returned error can't find the container with id 5a37868f2acadc031c6932af5c4e34ce77331aafc54d66c32b73455cbea08ef0 Oct 29 22:57:07.635023 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:07.634921 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86"] Oct 29 22:57:08.000738 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.000703 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86" event={"ID":"8c859b9a-7e01-4662-8fde-84242e0c387d","Type":"ContainerStarted","Data":"b4bf150559f75a9bc4e4e24d5eba4e2f49caeb1ef004bdfdf222068ae5cdd7c5"} Oct 29 22:57:08.001178 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.000759 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86" event={"ID":"8c859b9a-7e01-4662-8fde-84242e0c387d","Type":"ContainerStarted","Data":"5a37868f2acadc031c6932af5c4e34ce77331aafc54d66c32b73455cbea08ef0"} Oct 29 22:57:08.001178 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.000784 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86" Oct 29 22:57:08.002494 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.002459 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72" event={"ID":"478131d3-996c-4d45-ab02-52cf354b8de9","Type":"ContainerStarted","Data":"416fea6c69d5c8d26d25a6da0430db3d1619f7b1297dec0b84a9d0d062be9750"} Oct 29 22:57:08.002603 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.002509 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72" event={"ID":"478131d3-996c-4d45-ab02-52cf354b8de9","Type":"ContainerStarted","Data":"ee04797d476f7091093d89217ef16408daa8d6c7dcc507dc69426514d3cff173"} Oct 29 22:57:08.002603 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.002540 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72" Oct 29 22:57:08.012086 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.012064 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj"] Oct 29 22:57:08.016494 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.016473 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.034972 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.034948 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq"] Oct 29 22:57:08.040001 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.039978 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.043808 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.043770 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86" podStartSLOduration=2.043759117 podStartE2EDuration="2.043759117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:08.040412056 +0000 UTC m=+881.980959967" watchObservedRunningTime="2026-10-29 22:57:08.043759117 +0000 UTC m=+881.984307027" Oct 29 22:57:08.058183 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.058155 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj"] Oct 29 22:57:08.065365 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.065301 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.065496 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.065413 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.065543 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.065512 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.065597 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.065567 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.065655 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.065611 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.065738 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.065716 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/240bea83-173e-4b4a-9989-9608028afac4-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.065807 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.065789 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/240bea83-173e-4b4a-9989-9608028afac4-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.065868 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.065837 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.065926 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.065887 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.065926 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.065914 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/240bea83-173e-4b4a-9989-9608028afac4-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.066038 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.065973 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.066038 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.066006 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.066138 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.066064 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.066138 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.066079 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.066269 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.066175 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt5v8\" (UniqueName: \"kubernetes.io/projected/240bea83-173e-4b4a-9989-9608028afac4-kube-api-access-pt5v8\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.066269 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.066243 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsmvg\" (UniqueName: \"kubernetes.io/projected/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-kube-api-access-dsmvg\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.066369 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.066297 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.066369 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.066319 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.071749 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.071727 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq"] Oct 29 22:57:08.136088 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.136037 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72" podStartSLOduration=2.136022727 podStartE2EDuration="2.136022727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:08.130990883 +0000 UTC m=+882.071538795" watchObservedRunningTime="2026-10-29 22:57:08.136022727 +0000 UTC m=+882.076570637" Oct 29 22:57:08.167678 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.167646 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.167678 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.167683 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.168063 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.167728 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.168063 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.167742 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.168063 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.167762 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.168063 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.167808 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.168063 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.167840 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.168063 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.167875 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/240bea83-173e-4b4a-9989-9608028afac4-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.168063 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.167931 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/240bea83-173e-4b4a-9989-9608028afac4-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.168063 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.167966 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.168063 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.168010 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.168063 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.168038 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/240bea83-173e-4b4a-9989-9608028afac4-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.172059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.168099 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.172059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.168135 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.172059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.168227 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.172059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.168247 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.172059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.168252 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.172059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.168292 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.172059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.168424 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt5v8\" (UniqueName: \"kubernetes.io/projected/240bea83-173e-4b4a-9989-9608028afac4-kube-api-access-pt5v8\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.172059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.168464 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsmvg\" (UniqueName: \"kubernetes.io/projected/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-kube-api-access-dsmvg\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.172059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.168517 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.172059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.168524 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.172059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.168830 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.172059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.168984 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.172059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.169785 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/240bea83-173e-4b4a-9989-9608028afac4-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.172059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.170905 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.172059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.171076 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.172059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.171373 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.172059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.171426 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.172059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.171960 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.172980 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.172795 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.181787 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.181764 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/240bea83-173e-4b4a-9989-9608028afac4-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.192404 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.192376 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt5v8\" (UniqueName: \"kubernetes.io/projected/240bea83-173e-4b4a-9989-9608028afac4-kube-api-access-pt5v8\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.192530 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.192408 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/240bea83-173e-4b4a-9989-9608028afac4-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.194426 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.194339 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.194942 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.194914 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsmvg\" (UniqueName: \"kubernetes.io/projected/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-kube-api-access-dsmvg\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.335163 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.335068 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:08.370952 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.370923 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:08.673651 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.673623 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj"] Oct 29 22:57:08.674135 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:57:08.674113 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod240bea83_173e_4b4a_9989_9608028afac4.slice/crio-4e1284d036cfba76a80582169854337fa14bdfcd626178e37c79b277d802e1b3 WatchSource:0}: Error finding container 4e1284d036cfba76a80582169854337fa14bdfcd626178e37c79b277d802e1b3: Status 404 returned error can't find the container with id 4e1284d036cfba76a80582169854337fa14bdfcd626178e37c79b277d802e1b3 Oct 29 22:57:08.676853 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.676823 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:57:08.676922 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.676888 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:57:08.676922 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.676918 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:57:08.725076 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.725050 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq"] Oct 29 22:57:08.739298 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:57:08.739261 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f99e4c4_51fa_4e77_b863_c8867c9a08c2.slice/crio-8a8f66b1f2faf32dfcae36cf0fabda08d958004929f1e6ddef7af09540f4beac WatchSource:0}: Error finding container 8a8f66b1f2faf32dfcae36cf0fabda08d958004929f1e6ddef7af09540f4beac: Status 404 returned error can't find the container with id 8a8f66b1f2faf32dfcae36cf0fabda08d958004929f1e6ddef7af09540f4beac Oct 29 22:57:08.742155 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.742122 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:57:08.742254 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.742186 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:57:08.742254 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:08.742241 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:57:09.031083 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:09.030985 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" event={"ID":"3f99e4c4-51fa-4e77-b863-c8867c9a08c2","Type":"ContainerStarted","Data":"7dba42270b7085c57d0ab469137137f199031bbc0e964f1fbc924dd0c21834c2"} Oct 29 22:57:09.031681 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:09.031124 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" event={"ID":"3f99e4c4-51fa-4e77-b863-c8867c9a08c2","Type":"ContainerStarted","Data":"8a8f66b1f2faf32dfcae36cf0fabda08d958004929f1e6ddef7af09540f4beac"} Oct 29 22:57:09.036818 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:09.036784 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" event={"ID":"240bea83-173e-4b4a-9989-9608028afac4","Type":"ContainerStarted","Data":"bce99867a5f93185cc58d3235746788a460b32bc2e8ee471e53bf72856f6eb2f"} Oct 29 22:57:09.036966 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:09.036839 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" event={"ID":"240bea83-173e-4b4a-9989-9608028afac4","Type":"ContainerStarted","Data":"4e1284d036cfba76a80582169854337fa14bdfcd626178e37c79b277d802e1b3"} Oct 29 22:57:09.078037 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:09.077907 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" podStartSLOduration=1.077888094 podStartE2EDuration="1.077888094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:09.077789989 +0000 UTC m=+883.018337899" watchObservedRunningTime="2026-10-29 22:57:09.077888094 +0000 UTC m=+883.018436005" Oct 29 22:57:09.129553 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:09.129500 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" podStartSLOduration=2.129484152 podStartE2EDuration="2.129484152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:09.123579331 +0000 UTC m=+883.064127243" watchObservedRunningTime="2026-10-29 22:57:09.129484152 +0000 UTC m=+883.070032061" Oct 29 22:57:09.335764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:09.335682 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:09.337642 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:09.337609 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:57:09.337788 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:09.337683 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" podUID="240bea83-173e-4b4a-9989-9608028afac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 29 22:57:09.371482 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:09.371456 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:09.372890 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:09.372866 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:57:09.373009 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:09.372920 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" podUID="3f99e4c4-51fa-4e77-b863-c8867c9a08c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 29 22:57:09.923226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:09.919262 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb"] Oct 29 22:57:10.069225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:09.992821 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb" Oct 29 22:57:10.069225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:10.009149 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-cd525f\"" Oct 29 22:57:10.089262 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:10.079287 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-cd525f-dockercfg-tkxgt\"" Oct 29 22:57:10.089262 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:10.085271 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb"] Oct 29 22:57:10.158267 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:10.133531 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a8afe32-2de7-4a50-aced-ce5bf49c2db6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb\" (UID: \"6a8afe32-2de7-4a50-aced-ce5bf49c2db6\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb" Oct 29 22:57:10.158826 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:10.158774 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdv2d\" (UniqueName: \"kubernetes.io/projected/6a8afe32-2de7-4a50-aced-ce5bf49c2db6-kube-api-access-jdv2d\") pod \"payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb\" (UID: \"6a8afe32-2de7-4a50-aced-ce5bf49c2db6\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb" Oct 29 22:57:10.303307 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:10.295718 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a8afe32-2de7-4a50-aced-ce5bf49c2db6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb\" (UID: \"6a8afe32-2de7-4a50-aced-ce5bf49c2db6\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb" Oct 29 22:57:10.303307 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:10.303119 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdv2d\" (UniqueName: \"kubernetes.io/projected/6a8afe32-2de7-4a50-aced-ce5bf49c2db6-kube-api-access-jdv2d\") pod \"payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb\" (UID: \"6a8afe32-2de7-4a50-aced-ce5bf49c2db6\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb" Oct 29 22:57:10.303307 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:10.303242 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a8afe32-2de7-4a50-aced-ce5bf49c2db6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb\" (UID: \"6a8afe32-2de7-4a50-aced-ce5bf49c2db6\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb" Oct 29 22:57:10.350225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:10.349693 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:57:10.350225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:10.349758 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" podUID="240bea83-173e-4b4a-9989-9608028afac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 29 22:57:10.374227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:10.371919 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:57:10.374227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:10.371979 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" podUID="3f99e4c4-51fa-4e77-b863-c8867c9a08c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 29 22:57:10.385985 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:10.385948 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdv2d\" (UniqueName: \"kubernetes.io/projected/6a8afe32-2de7-4a50-aced-ce5bf49c2db6-kube-api-access-jdv2d\") pod \"payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb\" (UID: \"6a8afe32-2de7-4a50-aced-ce5bf49c2db6\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb" Oct 29 22:57:10.413047 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:10.413025 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb" Oct 29 22:57:10.979247 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:10.979219 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb"] Oct 29 22:57:11.168908 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:11.168872 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb" event={"ID":"6a8afe32-2de7-4a50-aced-ce5bf49c2db6","Type":"ContainerStarted","Data":"58ec7ce333236d4d0a94555e79db66e7e5a6aab2a7b662511790908fd1b17ba6"} Oct 29 22:57:11.338692 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:11.338593 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:57:11.339697 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:11.339665 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" podUID="240bea83-173e-4b4a-9989-9608028afac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 29 22:57:11.372420 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:11.372395 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:57:11.372654 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:11.372630 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" podUID="3f99e4c4-51fa-4e77-b863-c8867c9a08c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 29 22:57:12.196267 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:12.196187 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb" event={"ID":"6a8afe32-2de7-4a50-aced-ce5bf49c2db6","Type":"ContainerStarted","Data":"32176ce337ac01678e66ccd6e38be1ae7321dd5d540668443bde83a5f0723a6a"} Oct 29 22:57:12.197766 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:12.197738 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb" Oct 29 22:57:12.338224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:12.337626 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:57:12.338224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:12.337691 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" podUID="240bea83-173e-4b4a-9989-9608028afac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 29 22:57:12.386226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:12.374597 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:57:12.386226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:12.374671 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" podUID="3f99e4c4-51fa-4e77-b863-c8867c9a08c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 29 22:57:13.351227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:13.336684 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:57:13.351227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:13.336758 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" podUID="240bea83-173e-4b4a-9989-9608028afac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 29 22:57:13.373377 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:13.371670 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:57:13.373377 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:13.371739 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" podUID="3f99e4c4-51fa-4e77-b863-c8867c9a08c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 29 22:57:14.351662 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:14.351622 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:57:14.352282 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:14.352248 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" podUID="240bea83-173e-4b4a-9989-9608028afac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 29 22:57:14.371858 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:14.371649 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:57:14.371858 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:14.371704 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" podUID="3f99e4c4-51fa-4e77-b863-c8867c9a08c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 29 22:57:15.335932 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:15.335897 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 29 22:57:15.336110 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:15.335966 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" podUID="240bea83-173e-4b4a-9989-9608028afac4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 29 22:57:15.371687 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:15.371648 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:57:15.372096 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:15.371708 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" podUID="3f99e4c4-51fa-4e77-b863-c8867c9a08c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 29 22:57:16.376393 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:16.375001 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 29 22:57:16.376393 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:16.375063 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" podUID="3f99e4c4-51fa-4e77-b863-c8867c9a08c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 29 22:57:17.145224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:17.142532 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:17.145224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:17.144137 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:17.159911 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:17.159877 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 22:57:17.246721 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:17.246652 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb" podStartSLOduration=8.246627355 podStartE2EDuration="8.246627355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:12.261529854 +0000 UTC m=+886.202077767" watchObservedRunningTime="2026-10-29 22:57:17.246627355 +0000 UTC m=+891.187175265" Oct 29 22:57:17.404279 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:17.403378 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:17.405229 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:17.404890 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:17.412893 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:17.412866 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 22:57:19.059233 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:19.055430 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72" Oct 29 22:57:19.059233 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:19.057269 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86" Oct 29 22:57:19.388343 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:19.388293 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:19.707314 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:19.641708 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:19.710359 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:19.710327 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 22:57:19.916274 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:19.916229 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:19.937305 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:19.937250 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:19.973706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:19.954679 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Oct 29 22:57:19.973706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:19.954778 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:20.036348 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:20.036295 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:20.036873 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:20.036468 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" podUID="cb05c692-f4a5-49bf-a84d-ca3f03edd407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 29 22:57:20.037905 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:20.036551 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 29 22:57:20.338733 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:20.337913 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:20.338733 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:20.338251 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded" Oct 29 22:57:20.390620 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:20.389339 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 29 22:57:20.411230 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:20.400610 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:57:20.468670 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:20.468629 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:20.569680 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:20.569636 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:20.700013 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:20.698324 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:20.700013 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:20.698784 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:57:20.710925 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:20.710279 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:20.746227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:20.721155 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:20.781243 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:20.781191 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:57:20.934229 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:20.932887 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 29 22:57:20.934229 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:20.933449 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:57:20.981096 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:20.980999 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 22:57:20.983982 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:20.982963 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:57:21.059761 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:21.059722 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 29 22:57:21.073407 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:21.064417 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:21.073407 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:21.070588 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:57:21.073407 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:21.071371 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:57:21.073407 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:21.071413 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:57:21.081234 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:21.078111 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea5e601-k8zn9"] Oct 29 22:57:21.091275 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:21.091248 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea5e601-k8zn9"] Oct 29 22:57:21.191761 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:21.191294 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:21.328283 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:21.316482 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:21.328283 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:21.326854 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:57:21.358692 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:21.346772 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 29 22:57:21.358692 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:21.347158 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:57:21.515749 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:21.515698 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:21.530801 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:21.518159 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:57:21.595373 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:21.572528 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:21.595373 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:21.573385 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:57:21.624237 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:21.620346 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:21.839228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:21.821420 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 29 22:57:21.839228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:21.822546 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:57:22.095229 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:22.094863 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:22.097562 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:22.097527 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 29 22:57:22.129462 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:22.129416 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:22.155989 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:22.131875 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:22.235225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:22.132018 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:22.254234 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:22.245681 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:22.288746 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:22.288687 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="867af907-22c0-4265-bc64-576cc553de73" path="/var/lib/kubelet/pods/867af907-22c0-4265-bc64-576cc553de73/volumes" Oct 29 22:57:22.289462 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:22.289441 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:57:22.317229 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:22.298082 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:57:22.317229 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:22.298144 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:57:22.319674 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:22.319635 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:22.351070 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:22.350975 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:22.433447 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:22.433407 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:57:22.442406 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:22.442367 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:57:22.504481 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:22.504449 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:57:22.521828 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:22.521752 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:22.582706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:22.582677 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:57:22.620942 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:22.620890 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:22.664767 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:22.662738 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:57:22.688059 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:22.688030 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:57:22.800415 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:22.799298 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:57:22.800885 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:22.800864 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:57:22.867136 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:22.867095 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 22:57:22.895941 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:22.895866 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:57:23.209453 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:23.209374 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb" Oct 29 22:57:23.889697 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:57:23.889657 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:57:30.436107 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.436071 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p"] Oct 29 22:57:30.442780 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.442754 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.458729 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.458507 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a0dbf5-openshift-default-dockercfg-8q2hn\"" Oct 29 22:57:30.490071 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.490040 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p"] Oct 29 22:57:30.597192 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.597161 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-envoy\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.597383 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.597334 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-data\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.597451 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.597400 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-workload-socket\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.597509 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.597489 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-token\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.597573 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.597533 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-podinfo\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.597710 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.597693 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-credential-socket\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.597803 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.597789 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvw79\" (UniqueName: \"kubernetes.io/projected/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-kube-api-access-xvw79\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.597892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.597813 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-workload-certs\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.597892 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.597832 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istiod-ca-cert\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.706352 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.706248 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-workload-socket\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.708094 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.708067 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-token\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.708669 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.708644 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-podinfo\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.709472 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.709451 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-credential-socket\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.710016 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.709996 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvw79\" (UniqueName: \"kubernetes.io/projected/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-kube-api-access-xvw79\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.710450 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.710431 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-workload-certs\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.710911 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.710890 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istiod-ca-cert\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.711336 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.711316 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-envoy\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.726607 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.710833 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-workload-certs\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.726607 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.706981 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-workload-socket\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.726607 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.709867 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-credential-socket\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.727066 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.727041 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-data\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.728258 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.727792 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-podinfo\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.728488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.728018 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-data\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.728628 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.728437 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istiod-ca-cert\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.729983 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.729959 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-envoy\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.761222 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.758969 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvw79\" (UniqueName: \"kubernetes.io/projected/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-kube-api-access-xvw79\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.761222 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.759463 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-token\") pod \"e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:30.767634 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:30.766444 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:31.197329 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:57:31.197298 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf138dd09_a52b_4a74_bb6d_ffd90ca84f5c.slice/crio-4e864f61ad4a69839f2e50f5c76025d31cecb696dab1f4627b1bd30b9a7acb43 WatchSource:0}: Error finding container 4e864f61ad4a69839f2e50f5c76025d31cecb696dab1f4627b1bd30b9a7acb43: Status 404 returned error can't find the container with id 4e864f61ad4a69839f2e50f5c76025d31cecb696dab1f4627b1bd30b9a7acb43 Oct 29 22:57:31.199630 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:31.199600 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p"] Oct 29 22:57:31.200504 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:31.200472 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:57:31.200589 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:31.200534 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:57:31.200589 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:31.200565 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:57:31.368053 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:31.368017 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t"] Oct 29 22:57:31.373520 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:31.373487 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" Oct 29 22:57:31.378619 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:31.378588 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" event={"ID":"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c","Type":"ContainerStarted","Data":"6f6549fdf04cd4065afd1a2034a9a1e24a16a6407e164ab8900d24fbe7b9cd18"} Oct 29 22:57:31.378783 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:31.378646 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" event={"ID":"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c","Type":"ContainerStarted","Data":"4e864f61ad4a69839f2e50f5c76025d31cecb696dab1f4627b1bd30b9a7acb43"} Oct 29 22:57:31.387951 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:31.387925 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-27d16a-serving-cert\"" Oct 29 22:57:31.395294 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:31.395267 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t"] Oct 29 22:57:31.450296 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:31.442638 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-metrics-certs\") pod \"maas-api-e2e-api-b-27d16a-5c774b769f-76z4t\" (UID: \"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" Oct 29 22:57:31.450296 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:31.442718 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-maas-api-tls\") pod \"maas-api-e2e-api-b-27d16a-5c774b769f-76z4t\" (UID: \"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" Oct 29 22:57:31.450296 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:31.442823 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg97d\" (UniqueName: \"kubernetes.io/projected/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-kube-api-access-xg97d\") pod \"maas-api-e2e-api-b-27d16a-5c774b769f-76z4t\" (UID: \"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" Oct 29 22:57:31.450296 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:31.443135 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-openshift-service-ca\") pod \"maas-api-e2e-api-b-27d16a-5c774b769f-76z4t\" (UID: \"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" Oct 29 22:57:31.470564 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:31.470487 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" podStartSLOduration=1.470467075 podStartE2EDuration="1.470467075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:31.458056382 +0000 UTC m=+905.398604293" watchObservedRunningTime="2026-10-29 22:57:31.470467075 +0000 UTC m=+905.411014985" Oct 29 22:57:31.544717 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:31.544682 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-openshift-service-ca\") pod \"maas-api-e2e-api-b-27d16a-5c774b769f-76z4t\" (UID: \"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" Oct 29 22:57:31.544926 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:31.544745 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-metrics-certs\") pod \"maas-api-e2e-api-b-27d16a-5c774b769f-76z4t\" (UID: \"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" Oct 29 22:57:31.544926 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:31.544781 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-maas-api-tls\") pod \"maas-api-e2e-api-b-27d16a-5c774b769f-76z4t\" (UID: \"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" Oct 29 22:57:31.544926 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:31.544838 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg97d\" (UniqueName: \"kubernetes.io/projected/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-kube-api-access-xg97d\") pod \"maas-api-e2e-api-b-27d16a-5c774b769f-76z4t\" (UID: \"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" Oct 29 22:57:31.545092 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:57:31.544954 4821 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-serving-cert: secret "maas-api-e2e-api-b-27d16a-serving-cert" not found Oct 29 22:57:31.545092 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:57:31.545025 4821 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-maas-api-tls podName:d5a6b1fc-4fb1-41ae-99e2-359d9274c11d nodeName:}" failed. No retries permitted until 2026-10-29 22:57:32.04500424 +0000 UTC m=+905.985552145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-maas-api-tls") pod "maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" (UID: "d5a6b1fc-4fb1-41ae-99e2-359d9274c11d") : secret "maas-api-e2e-api-b-27d16a-serving-cert" not found Oct 29 22:57:31.545613 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:31.545581 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-openshift-service-ca\") pod \"maas-api-e2e-api-b-27d16a-5c774b769f-76z4t\" (UID: \"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" Oct 29 22:57:31.547567 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:31.547546 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-metrics-certs\") pod \"maas-api-e2e-api-b-27d16a-5c774b769f-76z4t\" (UID: \"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" Oct 29 22:57:31.570800 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:31.570769 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg97d\" (UniqueName: \"kubernetes.io/projected/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-kube-api-access-xg97d\") pod \"maas-api-e2e-api-b-27d16a-5c774b769f-76z4t\" (UID: \"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" Oct 29 22:57:31.768797 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:31.768710 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:32.053164 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:32.053081 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-maas-api-tls\") pod \"maas-api-e2e-api-b-27d16a-5c774b769f-76z4t\" (UID: \"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" Oct 29 22:57:32.055750 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:32.055720 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-maas-api-tls\") pod \"maas-api-e2e-api-b-27d16a-5c774b769f-76z4t\" (UID: \"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" Oct 29 22:57:32.293568 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:32.293526 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" Oct 29 22:57:32.701675 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:32.701631 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t"] Oct 29 22:57:32.778520 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:32.778491 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:33.398268 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:33.398153 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" event={"ID":"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d","Type":"ContainerStarted","Data":"ed35dc0a7c7ca1c3aa28e6243f0cd0040166ee9d369e5abe2ac1c610ed10be8b"} Oct 29 22:57:33.398268 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:33.398236 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" event={"ID":"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d","Type":"ContainerStarted","Data":"b539c5c98d7ba3a3836b68a4ec357ac53042d204bd2d8d3a875325dab5176b7c"} Oct 29 22:57:33.398268 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:33.398266 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" Oct 29 22:57:33.398719 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:33.398698 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:33.399627 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:33.399606 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:57:33.443050 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:33.443000 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" podStartSLOduration=2.066243552 podStartE2EDuration="2.442985985s" totalImagesPullingTime="376.742433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:33.438826769 +0000 UTC m=+907.379374678" watchObservedRunningTime="2026-10-29 22:57:33.442985985 +0000 UTC m=+907.383533894" Oct 29 22:57:35.053678 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:35.053645 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk"] Oct 29 22:57:35.058094 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:35.058074 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk" Oct 29 22:57:35.100259 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:35.088418 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-cd525f-serving-cert\"" Oct 29 22:57:35.100259 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:35.095098 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk"] Oct 29 22:57:35.216153 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:35.216114 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdbff57a-f530-441a-b708-8e3d2c64ed13-maas-api-tls\") pod \"maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk\" (UID: \"bdbff57a-f530-441a-b708-8e3d2c64ed13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk" Oct 29 22:57:35.216350 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:35.216172 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdbff57a-f530-441a-b708-8e3d2c64ed13-metrics-certs\") pod \"maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk\" (UID: \"bdbff57a-f530-441a-b708-8e3d2c64ed13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk" Oct 29 22:57:35.216420 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:35.216394 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ls2n\" (UniqueName: \"kubernetes.io/projected/bdbff57a-f530-441a-b708-8e3d2c64ed13-kube-api-access-5ls2n\") pod \"maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk\" (UID: \"bdbff57a-f530-441a-b708-8e3d2c64ed13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk" Oct 29 22:57:35.216539 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:35.216521 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdbff57a-f530-441a-b708-8e3d2c64ed13-openshift-service-ca\") pod \"maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk\" (UID: \"bdbff57a-f530-441a-b708-8e3d2c64ed13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk" Oct 29 22:57:35.317846 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:35.317754 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ls2n\" (UniqueName: \"kubernetes.io/projected/bdbff57a-f530-441a-b708-8e3d2c64ed13-kube-api-access-5ls2n\") pod \"maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk\" (UID: \"bdbff57a-f530-441a-b708-8e3d2c64ed13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk" Oct 29 22:57:35.318028 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:35.317872 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdbff57a-f530-441a-b708-8e3d2c64ed13-openshift-service-ca\") pod \"maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk\" (UID: \"bdbff57a-f530-441a-b708-8e3d2c64ed13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk" Oct 29 22:57:35.318028 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:35.317997 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdbff57a-f530-441a-b708-8e3d2c64ed13-maas-api-tls\") pod \"maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk\" (UID: \"bdbff57a-f530-441a-b708-8e3d2c64ed13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk" Oct 29 22:57:35.318156 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:35.318040 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdbff57a-f530-441a-b708-8e3d2c64ed13-metrics-certs\") pod \"maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk\" (UID: \"bdbff57a-f530-441a-b708-8e3d2c64ed13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk" Oct 29 22:57:35.318706 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:35.318678 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdbff57a-f530-441a-b708-8e3d2c64ed13-openshift-service-ca\") pod \"maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk\" (UID: \"bdbff57a-f530-441a-b708-8e3d2c64ed13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk" Oct 29 22:57:35.320876 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:35.320839 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdbff57a-f530-441a-b708-8e3d2c64ed13-metrics-certs\") pod \"maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk\" (UID: \"bdbff57a-f530-441a-b708-8e3d2c64ed13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk" Oct 29 22:57:35.321001 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:35.320943 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdbff57a-f530-441a-b708-8e3d2c64ed13-maas-api-tls\") pod \"maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk\" (UID: \"bdbff57a-f530-441a-b708-8e3d2c64ed13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk" Oct 29 22:57:35.340891 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:35.340861 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ls2n\" (UniqueName: \"kubernetes.io/projected/bdbff57a-f530-441a-b708-8e3d2c64ed13-kube-api-access-5ls2n\") pod \"maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk\" (UID: \"bdbff57a-f530-441a-b708-8e3d2c64ed13\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk" Oct 29 22:57:35.372622 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:35.372589 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk" Oct 29 22:57:35.699016 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:35.698988 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk"] Oct 29 22:57:35.699948 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:57:35.699916 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdbff57a_f530_441a_b708_8e3d2c64ed13.slice/crio-4fcf538b6804cab67b98e6eeb085b76eb795ad7f2028a09f5edaeb2525bf870f WatchSource:0}: Error finding container 4fcf538b6804cab67b98e6eeb085b76eb795ad7f2028a09f5edaeb2525bf870f: Status 404 returned error can't find the container with id 4fcf538b6804cab67b98e6eeb085b76eb795ad7f2028a09f5edaeb2525bf870f Oct 29 22:57:36.432288 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:36.432175 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk" event={"ID":"bdbff57a-f530-441a-b708-8e3d2c64ed13","Type":"ContainerStarted","Data":"19da036f3264d37334120b91f5302185fc7bf04a57cd068d67843504062acdb7"} Oct 29 22:57:36.432288 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:36.432261 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk" event={"ID":"bdbff57a-f530-441a-b708-8e3d2c64ed13","Type":"ContainerStarted","Data":"4fcf538b6804cab67b98e6eeb085b76eb795ad7f2028a09f5edaeb2525bf870f"} Oct 29 22:57:36.432750 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:36.432491 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk" Oct 29 22:57:36.478148 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:36.478096 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk" podStartSLOduration=1.041672803 podStartE2EDuration="1.478081052s" totalImagesPullingTime="436.408249ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:36.473560545 +0000 UTC m=+910.414108467" watchObservedRunningTime="2026-10-29 22:57:36.478081052 +0000 UTC m=+910.418628962" Oct 29 22:57:39.413601 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:39.413540 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" Oct 29 22:57:42.447717 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:42.447682 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk" Oct 29 22:57:51.307742 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:51.307706 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v"] Oct 29 22:57:51.312063 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:51.312045 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v" Oct 29 22:57:51.328223 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:51.328149 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-36befd-dockercfg-slzrj\"" Oct 29 22:57:51.338262 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:51.338232 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x54df\" (UniqueName: \"kubernetes.io/projected/799abc76-e641-4d92-b48f-d213c02c50ec-kube-api-access-x54df\") pod \"payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v\" (UID: \"799abc76-e641-4d92-b48f-d213c02c50ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v" Oct 29 22:57:51.338382 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:51.338279 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/799abc76-e641-4d92-b48f-d213c02c50ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v\" (UID: \"799abc76-e641-4d92-b48f-d213c02c50ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v" Oct 29 22:57:51.339673 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:51.339647 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-36befd\"" Oct 29 22:57:51.344986 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:51.344963 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v"] Oct 29 22:57:51.442280 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:51.440143 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/799abc76-e641-4d92-b48f-d213c02c50ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v\" (UID: \"799abc76-e641-4d92-b48f-d213c02c50ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v" Oct 29 22:57:51.442280 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:51.440573 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x54df\" (UniqueName: \"kubernetes.io/projected/799abc76-e641-4d92-b48f-d213c02c50ec-kube-api-access-x54df\") pod \"payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v\" (UID: \"799abc76-e641-4d92-b48f-d213c02c50ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v" Oct 29 22:57:51.442280 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:51.442101 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/799abc76-e641-4d92-b48f-d213c02c50ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v\" (UID: \"799abc76-e641-4d92-b48f-d213c02c50ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v" Oct 29 22:57:51.464926 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:51.464878 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x54df\" (UniqueName: \"kubernetes.io/projected/799abc76-e641-4d92-b48f-d213c02c50ec-kube-api-access-x54df\") pod \"payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v\" (UID: \"799abc76-e641-4d92-b48f-d213c02c50ec\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v" Oct 29 22:57:51.626676 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:51.626647 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v" Oct 29 22:57:52.004913 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:57:52.004873 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799abc76_e641_4d92_b48f_d213c02c50ec.slice/crio-4d8a430122f8ac3299a94b00b1f3488bbfcee725bcf53c8d4625c5b6e3fd92d5 WatchSource:0}: Error finding container 4d8a430122f8ac3299a94b00b1f3488bbfcee725bcf53c8d4625c5b6e3fd92d5: Status 404 returned error can't find the container with id 4d8a430122f8ac3299a94b00b1f3488bbfcee725bcf53c8d4625c5b6e3fd92d5 Oct 29 22:57:52.005409 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:52.005379 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v"] Oct 29 22:57:52.623442 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:52.623404 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v" event={"ID":"799abc76-e641-4d92-b48f-d213c02c50ec","Type":"ContainerStarted","Data":"fab2803dc653c374588d687ef379b76bd7df6897f1c05f23c5594c5c98a29def"} Oct 29 22:57:52.624493 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:52.623494 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v" event={"ID":"799abc76-e641-4d92-b48f-d213c02c50ec","Type":"ContainerStarted","Data":"4d8a430122f8ac3299a94b00b1f3488bbfcee725bcf53c8d4625c5b6e3fd92d5"} Oct 29 22:57:52.624493 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:52.623639 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v" Oct 29 22:57:52.665161 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:52.665111 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v" podStartSLOduration=1.665097863 podStartE2EDuration="1.665097863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:52.661952335 +0000 UTC m=+926.602500245" watchObservedRunningTime="2026-10-29 22:57:52.665097863 +0000 UTC m=+926.605645825" Oct 29 22:57:53.889262 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:57:53.889225 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:57:58.012227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:58.012089 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" podUID="cb05c692-f4a5-49bf-a84d-ca3f03edd407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 29 22:57:58.077227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:58.076298 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:58.133228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:58.132563 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:58.223238 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:58.223006 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:58.309919 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:58.308715 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 29 22:57:58.334224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:58.331265 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:58.358228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:58.356287 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 29 22:57:58.370397 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:58.370306 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sn4mm" podUID="bdeff50a-75a8-4272-9120-47c753cdcb5f" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.23:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:58.583407 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:58.583296 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:58.607433 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:58.607389 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:58.667431 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:58.667378 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:58.709222 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:58.707439 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:58.840446 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:58.840328 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:58.862017 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:58.861486 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 22:57:58.875412 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:58.874303 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:58.924459 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:58.902345 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:59.019234 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:59.019168 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" podUID="cb05c692-f4a5-49bf-a84d-ca3f03edd407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:59.119245 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:59.119216 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:57:59.250240 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:59.229165 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 29 22:57:59.254352 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:59.229757 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:57:59.254352 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:59.253856 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" podUID="e6f4fd46-5aab-49a8-9446-f71ed103d2eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:59.254352 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:59.229803 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:59.290566 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:59.269309 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:57:59.290566 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:59.269377 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:57:59.290566 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:59.273090 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 22:57:59.290566 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:59.273539 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:57:59.293325 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:59.293034 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:59.313707 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:59.312304 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:59.313707 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:59.312712 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:57:59.589320 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:59.576618 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 29 22:57:59.590368 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:59.590336 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:57:59.590612 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:59.580741 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:59.591001 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:59.590981 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:57:59.612339 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:59.591732 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:57:59.627610 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:59.615398 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:57:59.684225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:59.667800 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 29 22:57:59.684225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:59.677307 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:57:59.750229 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:59.726318 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 22:57:59.750229 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:57:59.727079 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:58:00.097229 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:00.092178 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 29 22:58:00.117714 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:00.113579 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:58:00.117714 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:00.115917 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 22:58:00.118436 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:00.118014 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:58:00.224864 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:00.224156 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" podUID="cb05c692-f4a5-49bf-a84d-ca3f03edd407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:00.335250 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:00.329332 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:00.359283 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:00.349163 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 29 22:58:00.378419 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:00.378348 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:00.380870 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:00.379476 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:00.412267 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:00.390618 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:00.621238 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:00.621107 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:00.621645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:00.621618 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 29 22:58:00.657125 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:00.657079 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:00.681257 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:00.662450 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:00.685233 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:00.682430 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 29 22:58:00.685596 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:00.685487 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 22:58:00.772097 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:00.772011 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:00.833259 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:00.831315 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:58:00.940960 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:00.940874 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 22:58:01.112225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:01.096292 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:01.138247 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:01.129830 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:01.250290 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:01.237328 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" podUID="cb05c692-f4a5-49bf-a84d-ca3f03edd407" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 29 22:58:01.382333 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:01.382284 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:01.423228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:01.384809 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 29 22:58:01.457853 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:01.391073 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 29 22:58:01.463087 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:01.407297 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:01.463651 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:01.433384 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 22:58:01.464859 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:01.464835 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 22:58:01.577438 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:01.577351 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 22:58:01.582592 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:01.582565 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 22:58:01.630039 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:01.622618 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:01.665359 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:01.664283 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:01.668383 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:01.668359 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 22:58:01.674576 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:01.674458 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:58:01.674576 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:01.674511 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" podUID="e438de1e-67a0-44a8-8ecd-e38ca050a541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:01.749925 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:01.749895 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 22:58:01.868615 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:01.868590 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 22:58:01.868859 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:01.867756 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:58:01.980084 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:01.980055 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 22:58:01.980386 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:01.980368 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 22:58:01.985696 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:01.985668 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 22:58:02.037952 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:02.023346 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 22:58:02.123057 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:02.122473 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 22:58:03.635846 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:03.635814 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v" Oct 29 22:58:06.840013 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:06.839974 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7608-c50acc"] Oct 29 22:58:06.845745 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:06.845721 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-c50acc" Oct 29 22:58:06.862246 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:06.862221 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7608-c50acc"] Oct 29 22:58:06.868352 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:06.868323 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fvr2l\"" Oct 29 22:58:06.897567 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:06.897533 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg6tz\" (UniqueName: \"kubernetes.io/projected/fc0227fe-61f9-410c-8ffe-5a2195abede4-kube-api-access-kg6tz\") pod \"test-curl-7608-c50acc\" (UID: \"fc0227fe-61f9-410c-8ffe-5a2195abede4\") " pod="opendatahub/test-curl-7608-c50acc" Oct 29 22:58:06.999403 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:06.999362 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg6tz\" (UniqueName: \"kubernetes.io/projected/fc0227fe-61f9-410c-8ffe-5a2195abede4-kube-api-access-kg6tz\") pod \"test-curl-7608-c50acc\" (UID: \"fc0227fe-61f9-410c-8ffe-5a2195abede4\") " pod="opendatahub/test-curl-7608-c50acc" Oct 29 22:58:07.024472 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:07.024440 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg6tz\" (UniqueName: \"kubernetes.io/projected/fc0227fe-61f9-410c-8ffe-5a2195abede4-kube-api-access-kg6tz\") pod \"test-curl-7608-c50acc\" (UID: \"fc0227fe-61f9-410c-8ffe-5a2195abede4\") " pod="opendatahub/test-curl-7608-c50acc" Oct 29 22:58:07.158339 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:07.158313 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-c50acc" Oct 29 22:58:07.476602 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:07.476574 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7608-c50acc"] Oct 29 22:58:07.476947 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:58:07.476924 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc0227fe_61f9_410c_8ffe_5a2195abede4.slice/crio-41ff23724227c1a9278712806a63c7c999450b0a7a98bbef52a50c127c5b466f WatchSource:0}: Error finding container 41ff23724227c1a9278712806a63c7c999450b0a7a98bbef52a50c127c5b466f: Status 404 returned error can't find the container with id 41ff23724227c1a9278712806a63c7c999450b0a7a98bbef52a50c127c5b466f Oct 29 22:58:08.151120 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:08.151080 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7608-c50acc" event={"ID":"fc0227fe-61f9-410c-8ffe-5a2195abede4","Type":"ContainerStarted","Data":"41ff23724227c1a9278712806a63c7c999450b0a7a98bbef52a50c127c5b466f"} Oct 29 22:58:10.173101 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.172969 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7608-c50acc" event={"ID":"fc0227fe-61f9-410c-8ffe-5a2195abede4","Type":"ContainerStarted","Data":"448a448be0624737fb5bd46c0017c3dbb2174ed9baaf38821d209b67cdb9edb7"} Oct 29 22:58:10.214270 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.214220 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7608-c50acc" podStartSLOduration=2.290182252 podStartE2EDuration="4.214188294s" totalImagesPullingTime="1.924006042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:10.209410526 +0000 UTC m=+944.149958438" watchObservedRunningTime="2026-10-29 22:58:10.214188294 +0000 UTC m=+944.154736204" Oct 29 22:58:10.546223 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.546129 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45"] Oct 29 22:58:10.550904 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.550881 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.579553 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.579515 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-27d16a-openshift-default-dockercfg-dq7sj\"" Oct 29 22:58:10.586738 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.586695 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45"] Oct 29 22:58:10.664465 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.664433 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjbpr\" (UniqueName: \"kubernetes.io/projected/525944e9-324d-4398-88bb-eeeea15478ce-kube-api-access-jjbpr\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.664465 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.664470 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-workload-certs\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.664733 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.664506 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-workload-socket\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.664733 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.664592 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-istio-envoy\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.664733 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.664663 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/525944e9-324d-4398-88bb-eeeea15478ce-istio-token\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.664733 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.664701 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-credential-socket\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.664936 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.664805 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-istio-data\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.664989 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.664930 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/525944e9-324d-4398-88bb-eeeea15478ce-istio-podinfo\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.665041 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.664996 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/525944e9-324d-4398-88bb-eeeea15478ce-istiod-ca-cert\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.766233 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.766184 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-istio-data\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.766400 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.766360 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/525944e9-324d-4398-88bb-eeeea15478ce-istio-podinfo\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.766469 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.766454 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/525944e9-324d-4398-88bb-eeeea15478ce-istiod-ca-cert\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.766563 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.766548 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjbpr\" (UniqueName: \"kubernetes.io/projected/525944e9-324d-4398-88bb-eeeea15478ce-kube-api-access-jjbpr\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.766598 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.766583 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-workload-certs\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.766649 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.766636 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-workload-socket\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.766699 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.766666 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-istio-envoy\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.766755 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.766719 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-istio-data\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.766811 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.766751 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/525944e9-324d-4398-88bb-eeeea15478ce-istio-token\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.766860 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.766810 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-credential-socket\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.767033 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.767005 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-workload-socket\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.767128 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.767054 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-workload-certs\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.767184 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.767162 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/525944e9-324d-4398-88bb-eeeea15478ce-istiod-ca-cert\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.767266 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.767187 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-credential-socket\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.769040 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.769018 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/525944e9-324d-4398-88bb-eeeea15478ce-istio-podinfo\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.769147 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.769127 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-istio-envoy\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.788862 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.788841 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/525944e9-324d-4398-88bb-eeeea15478ce-istio-token\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.789121 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.789101 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjbpr\" (UniqueName: \"kubernetes.io/projected/525944e9-324d-4398-88bb-eeeea15478ce-kube-api-access-jjbpr\") pod \"e2e-api-b-27d16a-openshift-default-869755f997-p2h45\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:10.866386 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:10.866356 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:11.216930 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:11.215165 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7608-c50acc"] Oct 29 22:58:11.279714 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:11.259599 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7608-c50acc"] Oct 29 22:58:11.285895 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:11.285863 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45"] Oct 29 22:58:11.300831 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:11.300312 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:58:11.300831 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:11.300409 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:58:11.300831 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:11.300442 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:58:12.269268 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:12.269174 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" event={"ID":"525944e9-324d-4398-88bb-eeeea15478ce","Type":"ContainerStarted","Data":"6ee2a527c5334137142f8768f3796a398a0b7798d68eb0d02b21faa44f1b9341"} Oct 29 22:58:12.269868 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:12.269325 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7608-c50acc" podUID="fc0227fe-61f9-410c-8ffe-5a2195abede4" containerName="test-curl-7608-c50acc" containerID="cri-o://448a448be0624737fb5bd46c0017c3dbb2174ed9baaf38821d209b67cdb9edb7" gracePeriod=2 Oct 29 22:58:12.272265 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:12.271295 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" event={"ID":"525944e9-324d-4398-88bb-eeeea15478ce","Type":"ContainerStarted","Data":"68ed6d9868a98e0946a938294cd97c6a3435988aede105d9272b0ba6ae7ea823"} Oct 29 22:58:12.359005 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:12.358939 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" podStartSLOduration=2.358920635 podStartE2EDuration="2.358920635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:12.329944627 +0000 UTC m=+946.270492538" watchObservedRunningTime="2026-10-29 22:58:12.358920635 +0000 UTC m=+946.299468594" Oct 29 22:58:12.866928 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:12.866890 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:12.868508 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:12.868482 4821 patch_prober.go:29] interesting pod/e2e-api-b-27d16a-openshift-default-869755f997-p2h45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:58:12.868630 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:12.868540 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" podUID="525944e9-324d-4398-88bb-eeeea15478ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 29 22:58:13.305111 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.305018 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv"] Oct 29 22:58:13.306632 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.306389 4821 cpu_manager.go:414] "containerMap: removing container" podUID="fc0227fe-61f9-410c-8ffe-5a2195abede4" containerName="test-curl-7608-c50acc" Oct 29 22:58:13.306823 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.306804 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc0227fe-61f9-410c-8ffe-5a2195abede4" containerName="test-curl-7608-c50acc" Oct 29 22:58:13.307278 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.307258 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc0227fe-61f9-410c-8ffe-5a2195abede4" containerName="test-curl-7608-c50acc" Oct 29 22:58:13.311734 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.311711 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.355455 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.355420 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-workload-socket\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.355841 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.355817 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-istio-data\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.355984 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.355967 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-credential-socket\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.356105 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.356091 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-workload-certs\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.356246 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.356229 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvbq4\" (UniqueName: \"kubernetes.io/projected/34564236-1496-484d-9fc9-8c35675b3dca-kube-api-access-rvbq4\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.356378 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.356363 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34564236-1496-484d-9fc9-8c35675b3dca-istio-token\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.356692 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.356670 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34564236-1496-484d-9fc9-8c35675b3dca-istiod-ca-cert\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.356843 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.356829 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-istio-envoy\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.356960 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.356947 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34564236-1496-484d-9fc9-8c35675b3dca-istio-podinfo\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.359254 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.359221 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0df9b0-openshift-default-dockercfg-fn9r7\"" Oct 29 22:58:13.361674 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.361484 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv"] Oct 29 22:58:13.366615 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.366570 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c"] Oct 29 22:58:13.372282 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.372262 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.388299 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.388266 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c"] Oct 29 22:58:13.496228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.494481 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-cd525f-openshift-default-dockercfg-rkhnr\"" Oct 29 22:58:13.511330 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.504337 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-workload-socket\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.511330 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.504399 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34564236-1496-484d-9fc9-8c35675b3dca-istiod-ca-cert\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.511330 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.504442 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdee46ad-6d49-4415-8d35-060d32549a59-istio-token\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.511330 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.504478 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-istio-envoy\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.511330 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.504511 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34564236-1496-484d-9fc9-8c35675b3dca-istio-podinfo\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.511330 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.504542 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-istio-data\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.511330 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.504598 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdee46ad-6d49-4415-8d35-060d32549a59-istiod-ca-cert\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.511330 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.504653 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-workload-socket\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.511330 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.504765 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-istio-data\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.511330 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.504808 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-credential-socket\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.511330 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.504841 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-workload-certs\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.511330 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.504978 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdee46ad-6d49-4415-8d35-060d32549a59-istio-podinfo\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.511330 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.505016 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvbq4\" (UniqueName: \"kubernetes.io/projected/34564236-1496-484d-9fc9-8c35675b3dca-kube-api-access-rvbq4\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.511330 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.505041 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-credential-socket\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.511330 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.505068 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpmt2\" (UniqueName: \"kubernetes.io/projected/bdee46ad-6d49-4415-8d35-060d32549a59-kube-api-access-gpmt2\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.511330 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.505106 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34564236-1496-484d-9fc9-8c35675b3dca-istio-token\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.511330 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.505167 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-istio-envoy\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.511330 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.505225 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-workload-certs\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.511330 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.506139 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34564236-1496-484d-9fc9-8c35675b3dca-istiod-ca-cert\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.511330 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.509144 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-istio-envoy\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.581279 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.580763 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-workload-certs\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.581279 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.581104 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-workload-socket\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.581496 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.581403 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-istio-data\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.609262 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.581643 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-credential-socket\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.609262 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.590704 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34564236-1496-484d-9fc9-8c35675b3dca-istio-podinfo\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.644221 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.638388 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-istio-data\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.644221 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.638466 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdee46ad-6d49-4415-8d35-060d32549a59-istiod-ca-cert\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.644221 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.638651 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdee46ad-6d49-4415-8d35-060d32549a59-istio-podinfo\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.644221 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.638687 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-credential-socket\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.644221 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.638719 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpmt2\" (UniqueName: \"kubernetes.io/projected/bdee46ad-6d49-4415-8d35-060d32549a59-kube-api-access-gpmt2\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.644221 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.638788 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-istio-envoy\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.644221 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.638823 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-workload-certs\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.644221 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.638942 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-workload-socket\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.644221 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.639000 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdee46ad-6d49-4415-8d35-060d32549a59-istio-token\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.696874 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.655280 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-istio-data\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.696874 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.655966 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdee46ad-6d49-4415-8d35-060d32549a59-istiod-ca-cert\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.696874 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.659233 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdee46ad-6d49-4415-8d35-060d32549a59-istio-podinfo\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.696874 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.659496 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-credential-socket\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.730352 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.701530 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-workload-certs\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.730352 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.701781 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-workload-socket\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.733192 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.733117 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-istio-envoy\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.780484 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.766889 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvbq4\" (UniqueName: \"kubernetes.io/projected/34564236-1496-484d-9fc9-8c35675b3dca-kube-api-access-rvbq4\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.791298 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.783458 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpmt2\" (UniqueName: \"kubernetes.io/projected/bdee46ad-6d49-4415-8d35-060d32549a59-kube-api-access-gpmt2\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.795933 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.791974 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdee46ad-6d49-4415-8d35-060d32549a59-istio-token\") pod \"e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:13.795933 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.794734 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34564236-1496-484d-9fc9-8c35675b3dca-istio-token\") pod \"e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:13.877244 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.874523 4821 patch_prober.go:29] interesting pod/e2e-api-b-27d16a-openshift-default-869755f997-p2h45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:58:13.877244 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.874580 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" podUID="525944e9-324d-4398-88bb-eeeea15478ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 29 22:58:13.988037 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:13.987745 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:14.015805 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:14.012266 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:14.464898 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:14.464743 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c"] Oct 29 22:58:14.468172 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:14.468130 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:58:14.468295 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:14.468244 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:58:14.468351 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:14.468293 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:58:14.482925 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:14.482895 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv"] Oct 29 22:58:14.493915 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:58:14.490119 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34564236_1496_484d_9fc9_8c35675b3dca.slice/crio-5b0f1c981ab5ea51c17bd69e10cac9573f0325c922585742b04fb22a13ebcac6 WatchSource:0}: Error finding container 5b0f1c981ab5ea51c17bd69e10cac9573f0325c922585742b04fb22a13ebcac6: Status 404 returned error can't find the container with id 5b0f1c981ab5ea51c17bd69e10cac9573f0325c922585742b04fb22a13ebcac6 Oct 29 22:58:14.493915 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:14.492904 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:58:14.493915 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:14.492991 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:58:14.493915 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:14.493032 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:58:14.589259 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:14.589230 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw"] Oct 29 22:58:14.652948 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:14.652852 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-c50acc" Oct 29 22:58:14.758308 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:14.749008 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc0227fe-61f9-410c-8ffe-5a2195abede4-kube-api-access-kg6tz\" (UniqueName: \"kubernetes.io/projected/fc0227fe-61f9-410c-8ffe-5a2195abede4-kube-api-access-kg6tz\") pod \"fc0227fe-61f9-410c-8ffe-5a2195abede4\" (UID: \"fc0227fe-61f9-410c-8ffe-5a2195abede4\") " Oct 29 22:58:14.779021 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:14.778982 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0227fe-61f9-410c-8ffe-5a2195abede4-kube-api-access-kg6tz" pod "fc0227fe-61f9-410c-8ffe-5a2195abede4" (UID: "fc0227fe-61f9-410c-8ffe-5a2195abede4"). InnerVolumeSpecName "kube-api-access-kg6tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:14.868526 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:14.868494 4821 patch_prober.go:29] interesting pod/e2e-api-b-27d16a-openshift-default-869755f997-p2h45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:58:14.884229 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:14.868786 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" podUID="525944e9-324d-4398-88bb-eeeea15478ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 29 22:58:14.905267 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:14.893017 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg6tz\" (UniqueName: \"kubernetes.io/projected/fc0227fe-61f9-410c-8ffe-5a2195abede4-kube-api-access-kg6tz\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:58:15.300619 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:15.300567 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" event={"ID":"34564236-1496-484d-9fc9-8c35675b3dca","Type":"ContainerStarted","Data":"d01cd83e27d2729b28c647d5632daf26ad1bbbba5cc498bcb284a9e97b9f3c66"} Oct 29 22:58:15.300821 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:15.300642 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" event={"ID":"34564236-1496-484d-9fc9-8c35675b3dca","Type":"ContainerStarted","Data":"5b0f1c981ab5ea51c17bd69e10cac9573f0325c922585742b04fb22a13ebcac6"} Oct 29 22:58:15.302264 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:15.302238 4821 generic.go:356] "Generic (PLEG): container finished" podID="fc0227fe-61f9-410c-8ffe-5a2195abede4" containerID="448a448be0624737fb5bd46c0017c3dbb2174ed9baaf38821d209b67cdb9edb7" exitCode=137 Oct 29 22:58:15.302421 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:15.302274 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-c50acc" Oct 29 22:58:15.302421 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:15.302312 4821 scope.go:122] "RemoveContainer" containerID="448a448be0624737fb5bd46c0017c3dbb2174ed9baaf38821d209b67cdb9edb7" Oct 29 22:58:15.304721 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:15.304691 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" event={"ID":"bdee46ad-6d49-4415-8d35-060d32549a59","Type":"ContainerStarted","Data":"4818ca980de99bfbc746c1b8eff4e3bc4c963cf6ae6f657e0931c28d426f8901"} Oct 29 22:58:15.304830 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:15.304747 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" event={"ID":"bdee46ad-6d49-4415-8d35-060d32549a59","Type":"ContainerStarted","Data":"3587c83a59559219c56236d8613373ac8cf2c2e1c958817f42127c0a0b90136c"} Oct 29 22:58:15.328724 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:15.322249 4821 scope.go:122] "RemoveContainer" containerID="448a448be0624737fb5bd46c0017c3dbb2174ed9baaf38821d209b67cdb9edb7" Oct 29 22:58:15.328724 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:58:15.322637 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"448a448be0624737fb5bd46c0017c3dbb2174ed9baaf38821d209b67cdb9edb7\": container with ID starting with 448a448be0624737fb5bd46c0017c3dbb2174ed9baaf38821d209b67cdb9edb7 not found: ID does not exist" containerID="448a448be0624737fb5bd46c0017c3dbb2174ed9baaf38821d209b67cdb9edb7" Oct 29 22:58:15.328724 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:15.322670 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"448a448be0624737fb5bd46c0017c3dbb2174ed9baaf38821d209b67cdb9edb7"} err="failed to get container status \"448a448be0624737fb5bd46c0017c3dbb2174ed9baaf38821d209b67cdb9edb7\": rpc error: code = NotFound desc = could not find container \"448a448be0624737fb5bd46c0017c3dbb2174ed9baaf38821d209b67cdb9edb7\": container with ID starting with 448a448be0624737fb5bd46c0017c3dbb2174ed9baaf38821d209b67cdb9edb7 not found: ID does not exist" Oct 29 22:58:15.357458 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:15.357395 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" podStartSLOduration=2.357372727 podStartE2EDuration="2.357372727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:15.353499273 +0000 UTC m=+949.294047185" watchObservedRunningTime="2026-10-29 22:58:15.357372727 +0000 UTC m=+949.297920643" Oct 29 22:58:15.366975 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:58:15.366637 4821 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7608-c50acc\" is forbidden: User \"system:node:ip-10-0-130-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-252.ec2.internal' and this object" podUID="fc0227fe-61f9-410c-8ffe-5a2195abede4" pod="opendatahub/test-curl-7608-c50acc" Oct 29 22:58:15.414262 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:15.413909 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" podStartSLOduration=2.413888517 podStartE2EDuration="2.413888517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:15.412150014 +0000 UTC m=+949.352697925" watchObservedRunningTime="2026-10-29 22:58:15.413888517 +0000 UTC m=+949.354436428" Oct 29 22:58:15.866896 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:15.866858 4821 patch_prober.go:29] interesting pod/e2e-api-b-27d16a-openshift-default-869755f997-p2h45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:58:15.867725 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:15.867691 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" podUID="525944e9-324d-4398-88bb-eeeea15478ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 29 22:58:16.010329 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:16.010298 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:16.060227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:16.055349 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:16.060227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:16.055553 4821 patch_prober.go:29] interesting pod/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:58:16.060227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:16.055598 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" podUID="bdee46ad-6d49-4415-8d35-060d32549a59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 29 22:58:16.090414 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:16.089860 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:16.206751 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:58:16.206594 4821 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7608-c50acc\" is forbidden: User \"system:node:ip-10-0-130-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-252.ec2.internal' and this object" podUID="fc0227fe-61f9-410c-8ffe-5a2195abede4" pod="opendatahub/test-curl-7608-c50acc" Oct 29 22:58:16.280944 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:58:16.280863 4821 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7608-c50acc\" is forbidden: User \"system:node:ip-10-0-130-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-252.ec2.internal' and this object" podUID="fc0227fe-61f9-410c-8ffe-5a2195abede4" pod="opendatahub/test-curl-7608-c50acc" Oct 29 22:58:16.393252 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:16.392501 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc0227fe-61f9-410c-8ffe-5a2195abede4" path="/var/lib/kubelet/pods/fc0227fe-61f9-410c-8ffe-5a2195abede4/volumes" Oct 29 22:58:16.400215 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:16.400174 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:16.401822 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:16.401802 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:58:16.870476 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:16.869926 4821 patch_prober.go:29] interesting pod/e2e-api-b-27d16a-openshift-default-869755f997-p2h45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:58:16.870476 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:16.869980 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" podUID="525944e9-324d-4398-88bb-eeeea15478ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 29 22:58:17.039227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:17.038597 4821 patch_prober.go:29] interesting pod/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:58:17.039227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:17.038662 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" podUID="bdee46ad-6d49-4415-8d35-060d32549a59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 29 22:58:17.868400 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:17.868316 4821 patch_prober.go:29] interesting pod/e2e-api-b-27d16a-openshift-default-869755f997-p2h45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 29 22:58:17.868400 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:17.868377 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" podUID="525944e9-324d-4398-88bb-eeeea15478ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 29 22:58:18.058595 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:18.058113 4821 patch_prober.go:29] interesting pod/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:58:18.058595 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:18.058175 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" podUID="bdee46ad-6d49-4415-8d35-060d32549a59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 29 22:58:18.872882 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:18.872844 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:18.873359 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:18.873330 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:18.874126 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:18.874110 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 22:58:19.051758 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:19.051728 4821 patch_prober.go:29] interesting pod/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:58:19.052378 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:19.052345 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" podUID="bdee46ad-6d49-4415-8d35-060d32549a59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 29 22:58:19.589583 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:19.589513 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw" podUID="8e11dce4-40a9-47bf-a689-b3b81d86417b" containerName="payload-processing" containerID="cri-o://507d523db49884bd713f2bc7b413002af7a66722615927bfedec1469632d0172" gracePeriod=25 Oct 29 22:58:20.041280 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:20.037626 4821 patch_prober.go:29] interesting pod/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:58:20.041280 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:20.037687 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" podUID="bdee46ad-6d49-4415-8d35-060d32549a59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 29 22:58:20.270572 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:20.270539 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw" Oct 29 22:58:20.361133 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:20.361091 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e11dce4-40a9-47bf-a689-b3b81d86417b-kube-api-access-dfzcm\" (UniqueName: \"kubernetes.io/projected/8e11dce4-40a9-47bf-a689-b3b81d86417b-kube-api-access-dfzcm\") pod \"8e11dce4-40a9-47bf-a689-b3b81d86417b\" (UID: \"8e11dce4-40a9-47bf-a689-b3b81d86417b\") " Oct 29 22:58:20.361384 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:20.361278 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e11dce4-40a9-47bf-a689-b3b81d86417b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e11dce4-40a9-47bf-a689-b3b81d86417b-plugins-config-volume\") pod \"8e11dce4-40a9-47bf-a689-b3b81d86417b\" (UID: \"8e11dce4-40a9-47bf-a689-b3b81d86417b\") " Oct 29 22:58:20.362902 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:20.362228 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e11dce4-40a9-47bf-a689-b3b81d86417b-plugins-config-volume" pod "8e11dce4-40a9-47bf-a689-b3b81d86417b" (UID: "8e11dce4-40a9-47bf-a689-b3b81d86417b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:58:20.364707 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:20.364663 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e11dce4-40a9-47bf-a689-b3b81d86417b-kube-api-access-dfzcm" pod "8e11dce4-40a9-47bf-a689-b3b81d86417b" (UID: "8e11dce4-40a9-47bf-a689-b3b81d86417b"). InnerVolumeSpecName "kube-api-access-dfzcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:20.462006 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:20.461974 4821 generic.go:356] "Generic (PLEG): container finished" podID="8e11dce4-40a9-47bf-a689-b3b81d86417b" containerID="507d523db49884bd713f2bc7b413002af7a66722615927bfedec1469632d0172" exitCode=0 Oct 29 22:58:20.463073 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:20.463046 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw" event={"ID":"8e11dce4-40a9-47bf-a689-b3b81d86417b","Type":"ContainerDied","Data":"507d523db49884bd713f2bc7b413002af7a66722615927bfedec1469632d0172"} Oct 29 22:58:20.463188 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:20.463102 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw" event={"ID":"8e11dce4-40a9-47bf-a689-b3b81d86417b","Type":"ContainerDied","Data":"a449827b764cd8c22e5016a3459e8762b985015d3e21817dbaf5113c70ea1b5e"} Oct 29 22:58:20.463188 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:20.463127 4821 scope.go:122] "RemoveContainer" containerID="507d523db49884bd713f2bc7b413002af7a66722615927bfedec1469632d0172" Oct 29 22:58:20.463358 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:20.463338 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw" Oct 29 22:58:20.464338 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:20.464321 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfzcm\" (UniqueName: \"kubernetes.io/projected/8e11dce4-40a9-47bf-a689-b3b81d86417b-kube-api-access-dfzcm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:58:20.464392 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:20.464341 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e11dce4-40a9-47bf-a689-b3b81d86417b-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:58:20.476381 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:20.476360 4821 scope.go:122] "RemoveContainer" containerID="507d523db49884bd713f2bc7b413002af7a66722615927bfedec1469632d0172" Oct 29 22:58:20.476648 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:58:20.476629 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"507d523db49884bd713f2bc7b413002af7a66722615927bfedec1469632d0172\": container with ID starting with 507d523db49884bd713f2bc7b413002af7a66722615927bfedec1469632d0172 not found: ID does not exist" containerID="507d523db49884bd713f2bc7b413002af7a66722615927bfedec1469632d0172" Oct 29 22:58:20.476719 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:20.476655 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507d523db49884bd713f2bc7b413002af7a66722615927bfedec1469632d0172"} err="failed to get container status \"507d523db49884bd713f2bc7b413002af7a66722615927bfedec1469632d0172\": rpc error: code = NotFound desc = could not find container \"507d523db49884bd713f2bc7b413002af7a66722615927bfedec1469632d0172\": container with ID starting with 507d523db49884bd713f2bc7b413002af7a66722615927bfedec1469632d0172 not found: ID does not exist" Oct 29 22:58:20.540872 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:20.540839 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw"] Oct 29 22:58:20.547501 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:20.547465 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e4a049-5bb5d44fb-86gkw"] Oct 29 22:58:21.033228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:21.030166 4821 patch_prober.go:29] interesting pod/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 29 22:58:21.033228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:21.030258 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" podUID="bdee46ad-6d49-4415-8d35-060d32549a59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 29 22:58:22.040755 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.040721 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:22.041346 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.041194 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:22.042564 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.042538 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 22:58:22.211513 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.211476 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e11dce4-40a9-47bf-a689-b3b81d86417b" path="/var/lib/kubelet/pods/8e11dce4-40a9-47bf-a689-b3b81d86417b/volumes" Oct 29 22:58:22.360674 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.360641 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d"] Oct 29 22:58:22.361329 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.361314 4821 cpu_manager.go:414] "containerMap: removing container" podUID="8e11dce4-40a9-47bf-a689-b3b81d86417b" containerName="payload-processing" Oct 29 22:58:22.361329 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.361329 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e11dce4-40a9-47bf-a689-b3b81d86417b" containerName="payload-processing" Oct 29 22:58:22.361429 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.361425 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e11dce4-40a9-47bf-a689-b3b81d86417b" containerName="payload-processing" Oct 29 22:58:22.365078 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.365059 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:22.379376 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.379352 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 29 22:58:22.379376 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.379367 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 29 22:58:22.379542 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.379497 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 29 22:58:22.379583 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.379569 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dpk7p\"" Oct 29 22:58:22.413729 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.413701 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d"] Oct 29 22:58:22.416128 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.416102 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b04ee4bb-9267-47f3-8ef0-d077400db137-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tf54d\" (UID: \"b04ee4bb-9267-47f3-8ef0-d077400db137\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:22.416296 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.416180 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwlgp\" (UniqueName: \"kubernetes.io/projected/b04ee4bb-9267-47f3-8ef0-d077400db137-kube-api-access-hwlgp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tf54d\" (UID: \"b04ee4bb-9267-47f3-8ef0-d077400db137\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:22.416296 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.416252 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b04ee4bb-9267-47f3-8ef0-d077400db137-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tf54d\" (UID: \"b04ee4bb-9267-47f3-8ef0-d077400db137\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:22.416404 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.416342 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b04ee4bb-9267-47f3-8ef0-d077400db137-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tf54d\" (UID: \"b04ee4bb-9267-47f3-8ef0-d077400db137\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:22.416458 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.416407 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b04ee4bb-9267-47f3-8ef0-d077400db137-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tf54d\" (UID: \"b04ee4bb-9267-47f3-8ef0-d077400db137\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:22.416514 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.416480 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04ee4bb-9267-47f3-8ef0-d077400db137-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tf54d\" (UID: \"b04ee4bb-9267-47f3-8ef0-d077400db137\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:22.416607 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.416588 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b04ee4bb-9267-47f3-8ef0-d077400db137-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tf54d\" (UID: \"b04ee4bb-9267-47f3-8ef0-d077400db137\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:22.518183 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.518148 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b04ee4bb-9267-47f3-8ef0-d077400db137-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tf54d\" (UID: \"b04ee4bb-9267-47f3-8ef0-d077400db137\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:22.518366 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.518274 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04ee4bb-9267-47f3-8ef0-d077400db137-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tf54d\" (UID: \"b04ee4bb-9267-47f3-8ef0-d077400db137\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:22.518416 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.518400 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b04ee4bb-9267-47f3-8ef0-d077400db137-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tf54d\" (UID: \"b04ee4bb-9267-47f3-8ef0-d077400db137\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:22.518533 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.518515 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b04ee4bb-9267-47f3-8ef0-d077400db137-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tf54d\" (UID: \"b04ee4bb-9267-47f3-8ef0-d077400db137\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:22.518658 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.518625 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwlgp\" (UniqueName: \"kubernetes.io/projected/b04ee4bb-9267-47f3-8ef0-d077400db137-kube-api-access-hwlgp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tf54d\" (UID: \"b04ee4bb-9267-47f3-8ef0-d077400db137\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:22.518750 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.518724 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b04ee4bb-9267-47f3-8ef0-d077400db137-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tf54d\" (UID: \"b04ee4bb-9267-47f3-8ef0-d077400db137\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:22.518825 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.518804 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b04ee4bb-9267-47f3-8ef0-d077400db137-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tf54d\" (UID: \"b04ee4bb-9267-47f3-8ef0-d077400db137\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:22.519329 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.519304 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b04ee4bb-9267-47f3-8ef0-d077400db137-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tf54d\" (UID: \"b04ee4bb-9267-47f3-8ef0-d077400db137\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:22.520998 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.520976 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b04ee4bb-9267-47f3-8ef0-d077400db137-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tf54d\" (UID: \"b04ee4bb-9267-47f3-8ef0-d077400db137\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:22.521483 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.521461 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b04ee4bb-9267-47f3-8ef0-d077400db137-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tf54d\" (UID: \"b04ee4bb-9267-47f3-8ef0-d077400db137\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:22.521577 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.521509 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b04ee4bb-9267-47f3-8ef0-d077400db137-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tf54d\" (UID: \"b04ee4bb-9267-47f3-8ef0-d077400db137\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:22.521577 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.521561 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b04ee4bb-9267-47f3-8ef0-d077400db137-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tf54d\" (UID: \"b04ee4bb-9267-47f3-8ef0-d077400db137\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:22.553311 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.547706 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b04ee4bb-9267-47f3-8ef0-d077400db137-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tf54d\" (UID: \"b04ee4bb-9267-47f3-8ef0-d077400db137\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:22.553311 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.548039 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwlgp\" (UniqueName: \"kubernetes.io/projected/b04ee4bb-9267-47f3-8ef0-d077400db137-kube-api-access-hwlgp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tf54d\" (UID: \"b04ee4bb-9267-47f3-8ef0-d077400db137\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:22.680141 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:22.680048 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:23.052591 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:23.052565 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d"] Oct 29 22:58:23.053335 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:58:23.053310 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04ee4bb_9267_47f3_8ef0_d077400db137.slice/crio-3ded21751222b30b59cef5bd447f409e5a7243705da28da1f4b780eebcd53bd1 WatchSource:0}: Error finding container 3ded21751222b30b59cef5bd447f409e5a7243705da28da1f4b780eebcd53bd1: Status 404 returned error can't find the container with id 3ded21751222b30b59cef5bd447f409e5a7243705da28da1f4b780eebcd53bd1 Oct 29 22:58:23.528353 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:23.528314 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" event={"ID":"b04ee4bb-9267-47f3-8ef0-d077400db137","Type":"ContainerStarted","Data":"3ded21751222b30b59cef5bd447f409e5a7243705da28da1f4b780eebcd53bd1"} Oct 29 22:58:23.908788 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:58:23.908731 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:58:25.918363 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:25.918332 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:58:25.918660 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:25.918401 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:58:26.369872 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:26.369835 4821 scope.go:122] "RemoveContainer" containerID="41f542a9ba547fd9edf6fd0eca81ac2b890495c811db97dd164fdd51fec95ba4" Oct 29 22:58:26.563973 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:26.563927 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" event={"ID":"b04ee4bb-9267-47f3-8ef0-d077400db137","Type":"ContainerStarted","Data":"d806d87267be29ed57534cd775c109ab11ceb85fe3682844c7852125714886cf"} Oct 29 22:58:26.564156 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:26.564059 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:26.634529 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:26.634420 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" podStartSLOduration=1.7730926249999999 podStartE2EDuration="4.634400923s" totalImagesPullingTime="2.861308298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:26.618877487 +0000 UTC m=+960.559425398" watchObservedRunningTime="2026-10-29 22:58:26.634400923 +0000 UTC m=+960.574948837" Oct 29 22:58:27.576414 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:27.576368 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tf54d" Oct 29 22:58:46.808974 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:46.808931 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5"] Oct 29 22:58:47.828112 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:47.828081 4821 generic.go:356] "Generic (PLEG): container finished" podID="f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" containerID="6f6549fdf04cd4065afd1a2034a9a1e24a16a6407e164ab8900d24fbe7b9cd18" exitCode=0 Oct 29 22:58:47.828772 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:47.828126 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" event={"ID":"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c","Type":"ContainerDied","Data":"6f6549fdf04cd4065afd1a2034a9a1e24a16a6407e164ab8900d24fbe7b9cd18"} Oct 29 22:58:47.828772 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:47.828631 4821 scope.go:122] "RemoveContainer" containerID="6f6549fdf04cd4065afd1a2034a9a1e24a16a6407e164ab8900d24fbe7b9cd18" Oct 29 22:58:47.830510 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:47.830479 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:58:47.830620 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:47.830552 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:58:47.830620 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:47.830582 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:58:48.398649 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:48.398611 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:58:48.838739 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:48.838650 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" event={"ID":"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c","Type":"ContainerStarted","Data":"e92edf67746ce7c833ab122bb9f7b193f9f64c176eebe63410eb34fc22549e8e"} Oct 29 22:58:49.768966 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:49.768930 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:58:49.770162 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:49.770133 4821 patch_prober.go:29] interesting pod/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 29 22:58:49.770319 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:49.770192 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" podUID="f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 29 22:58:50.510276 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:50.506636 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v"] Oct 29 22:58:50.777609 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:50.777328 4821 patch_prober.go:29] interesting pod/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 29 22:58:50.777609 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:50.777395 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" podUID="f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 29 22:58:51.769645 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:51.769611 4821 patch_prober.go:29] interesting pod/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 29 22:58:51.770036 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:51.769669 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" podUID="f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 29 22:58:51.809838 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:51.809802 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5" podUID="eea051e6-e43a-473a-9c98-1a018a8e3d27" containerName="payload-processing" containerID="cri-o://262493a48269796629dd147f0d447890f5ebfc1368d6944797b48c9983b6a28f" gracePeriod=25 Oct 29 22:58:52.267632 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:52.267607 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5" Oct 29 22:58:52.392093 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:52.392054 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eea051e6-e43a-473a-9c98-1a018a8e3d27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea051e6-e43a-473a-9c98-1a018a8e3d27-plugins-config-volume\") pod \"eea051e6-e43a-473a-9c98-1a018a8e3d27\" (UID: \"eea051e6-e43a-473a-9c98-1a018a8e3d27\") " Oct 29 22:58:52.392324 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:52.392239 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea051e6-e43a-473a-9c98-1a018a8e3d27-kube-api-access-7xbbb\" (UniqueName: \"kubernetes.io/projected/eea051e6-e43a-473a-9c98-1a018a8e3d27-kube-api-access-7xbbb\") pod \"eea051e6-e43a-473a-9c98-1a018a8e3d27\" (UID: \"eea051e6-e43a-473a-9c98-1a018a8e3d27\") " Oct 29 22:58:52.392545 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:52.392518 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea051e6-e43a-473a-9c98-1a018a8e3d27-plugins-config-volume" pod "eea051e6-e43a-473a-9c98-1a018a8e3d27" (UID: "eea051e6-e43a-473a-9c98-1a018a8e3d27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:58:52.392947 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:52.392929 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea051e6-e43a-473a-9c98-1a018a8e3d27-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:58:52.394873 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:52.394841 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea051e6-e43a-473a-9c98-1a018a8e3d27-kube-api-access-7xbbb" pod "eea051e6-e43a-473a-9c98-1a018a8e3d27" (UID: "eea051e6-e43a-473a-9c98-1a018a8e3d27"). InnerVolumeSpecName "kube-api-access-7xbbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:52.494862 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:52.494832 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xbbb\" (UniqueName: \"kubernetes.io/projected/eea051e6-e43a-473a-9c98-1a018a8e3d27-kube-api-access-7xbbb\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:58:52.769680 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:52.769598 4821 patch_prober.go:29] interesting pod/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 29 22:58:52.769680 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:52.769647 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" podUID="f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 29 22:58:52.879395 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:52.879358 4821 generic.go:356] "Generic (PLEG): container finished" podID="eea051e6-e43a-473a-9c98-1a018a8e3d27" containerID="262493a48269796629dd147f0d447890f5ebfc1368d6944797b48c9983b6a28f" exitCode=0 Oct 29 22:58:52.879598 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:52.879411 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5" Oct 29 22:58:52.879598 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:52.879453 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5" event={"ID":"eea051e6-e43a-473a-9c98-1a018a8e3d27","Type":"ContainerDied","Data":"262493a48269796629dd147f0d447890f5ebfc1368d6944797b48c9983b6a28f"} Oct 29 22:58:52.879598 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:52.879510 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5" event={"ID":"eea051e6-e43a-473a-9c98-1a018a8e3d27","Type":"ContainerDied","Data":"ed64ec60128ac4332d45d6404323e249a2c979c575bd23e5f290f42941b05990"} Oct 29 22:58:52.879598 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:52.879548 4821 scope.go:122] "RemoveContainer" containerID="262493a48269796629dd147f0d447890f5ebfc1368d6944797b48c9983b6a28f" Oct 29 22:58:52.891634 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:52.891612 4821 scope.go:122] "RemoveContainer" containerID="262493a48269796629dd147f0d447890f5ebfc1368d6944797b48c9983b6a28f" Oct 29 22:58:52.891921 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:58:52.891903 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"262493a48269796629dd147f0d447890f5ebfc1368d6944797b48c9983b6a28f\": container with ID starting with 262493a48269796629dd147f0d447890f5ebfc1368d6944797b48c9983b6a28f not found: ID does not exist" containerID="262493a48269796629dd147f0d447890f5ebfc1368d6944797b48c9983b6a28f" Oct 29 22:58:52.891990 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:52.891930 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"262493a48269796629dd147f0d447890f5ebfc1368d6944797b48c9983b6a28f"} err="failed to get container status \"262493a48269796629dd147f0d447890f5ebfc1368d6944797b48c9983b6a28f\": rpc error: code = NotFound desc = could not find container \"262493a48269796629dd147f0d447890f5ebfc1368d6944797b48c9983b6a28f\": container with ID starting with 262493a48269796629dd147f0d447890f5ebfc1368d6944797b48c9983b6a28f not found: ID does not exist" Oct 29 22:58:53.039285 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:53.038828 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5"] Oct 29 22:58:53.159225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:53.158298 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a37a29-85dddb468d-s6nk5"] Oct 29 22:58:53.768799 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:53.768764 4821 patch_prober.go:29] interesting pod/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 29 22:58:53.769072 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:53.768829 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" podUID="f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 29 22:58:53.895038 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:58:53.895000 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:58:54.171220 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:54.171177 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea051e6-e43a-473a-9c98-1a018a8e3d27" path="/var/lib/kubelet/pods/eea051e6-e43a-473a-9c98-1a018a8e3d27/volumes" Oct 29 22:58:54.829753 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:54.829717 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:58:54.830334 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:54.830312 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:58:54.831611 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:54.831583 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 22:58:55.507666 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:55.507613 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v" podUID="799abc76-e641-4d92-b48f-d213c02c50ec" containerName="payload-pre-processing" containerID="cri-o://fab2803dc653c374588d687ef379b76bd7df6897f1c05f23c5594c5c98a29def" gracePeriod=25 Oct 29 22:58:55.809818 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:55.809792 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v" Oct 29 22:58:55.909417 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:55.909383 4821 generic.go:356] "Generic (PLEG): container finished" podID="799abc76-e641-4d92-b48f-d213c02c50ec" containerID="fab2803dc653c374588d687ef379b76bd7df6897f1c05f23c5594c5c98a29def" exitCode=0 Oct 29 22:58:55.909630 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:55.909446 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v" Oct 29 22:58:55.909630 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:55.909480 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v" event={"ID":"799abc76-e641-4d92-b48f-d213c02c50ec","Type":"ContainerDied","Data":"fab2803dc653c374588d687ef379b76bd7df6897f1c05f23c5594c5c98a29def"} Oct 29 22:58:55.909630 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:55.909554 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v" event={"ID":"799abc76-e641-4d92-b48f-d213c02c50ec","Type":"ContainerDied","Data":"4d8a430122f8ac3299a94b00b1f3488bbfcee725bcf53c8d4625c5b6e3fd92d5"} Oct 29 22:58:55.909630 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:55.909588 4821 scope.go:122] "RemoveContainer" containerID="fab2803dc653c374588d687ef379b76bd7df6897f1c05f23c5594c5c98a29def" Oct 29 22:58:55.922054 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:55.922033 4821 scope.go:122] "RemoveContainer" containerID="fab2803dc653c374588d687ef379b76bd7df6897f1c05f23c5594c5c98a29def" Oct 29 22:58:55.922592 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:58:55.922469 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab2803dc653c374588d687ef379b76bd7df6897f1c05f23c5594c5c98a29def\": container with ID starting with fab2803dc653c374588d687ef379b76bd7df6897f1c05f23c5594c5c98a29def not found: ID does not exist" containerID="fab2803dc653c374588d687ef379b76bd7df6897f1c05f23c5594c5c98a29def" Oct 29 22:58:55.922592 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:55.922513 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab2803dc653c374588d687ef379b76bd7df6897f1c05f23c5594c5c98a29def"} err="failed to get container status \"fab2803dc653c374588d687ef379b76bd7df6897f1c05f23c5594c5c98a29def\": rpc error: code = NotFound desc = could not find container \"fab2803dc653c374588d687ef379b76bd7df6897f1c05f23c5594c5c98a29def\": container with ID starting with fab2803dc653c374588d687ef379b76bd7df6897f1c05f23c5594c5c98a29def not found: ID does not exist" Oct 29 22:58:55.989625 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:55.989598 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/799abc76-e641-4d92-b48f-d213c02c50ec-kube-api-access-x54df\" (UniqueName: \"kubernetes.io/projected/799abc76-e641-4d92-b48f-d213c02c50ec-kube-api-access-x54df\") pod \"799abc76-e641-4d92-b48f-d213c02c50ec\" (UID: \"799abc76-e641-4d92-b48f-d213c02c50ec\") " Oct 29 22:58:55.989813 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:55.989642 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/799abc76-e641-4d92-b48f-d213c02c50ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/799abc76-e641-4d92-b48f-d213c02c50ec-plugins-config-volume\") pod \"799abc76-e641-4d92-b48f-d213c02c50ec\" (UID: \"799abc76-e641-4d92-b48f-d213c02c50ec\") " Oct 29 22:58:55.990091 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:55.990059 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/799abc76-e641-4d92-b48f-d213c02c50ec-plugins-config-volume" pod "799abc76-e641-4d92-b48f-d213c02c50ec" (UID: "799abc76-e641-4d92-b48f-d213c02c50ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:58:55.990498 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:55.990476 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/799abc76-e641-4d92-b48f-d213c02c50ec-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:58:55.992265 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:55.992238 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/799abc76-e641-4d92-b48f-d213c02c50ec-kube-api-access-x54df" pod "799abc76-e641-4d92-b48f-d213c02c50ec" (UID: "799abc76-e641-4d92-b48f-d213c02c50ec"). InnerVolumeSpecName "kube-api-access-x54df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:56.093320 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:56.093186 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x54df\" (UniqueName: \"kubernetes.io/projected/799abc76-e641-4d92-b48f-d213c02c50ec-kube-api-access-x54df\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:58:56.249875 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:56.249834 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v"] Oct 29 22:58:56.262765 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:56.262730 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36befd-65d7d484c5-t954v"] Oct 29 22:58:58.172960 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:58.172922 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="799abc76-e641-4d92-b48f-d213c02c50ec" path="/var/lib/kubelet/pods/799abc76-e641-4d92-b48f-d213c02c50ec/volumes" Oct 29 22:58:58.362318 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:58.362276 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b"] Oct 29 22:58:58.362940 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:58:58.362570 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" podUID="c49af22e-9041-4f66-a3ff-175ad6d7b342" containerName="istio-proxy" containerID="cri-o://d2a8b524ca4f295ef4c469ea4eaf324974d9e43c62f9eaa134b81e92c2aa8a2c" gracePeriod=30 Oct 29 22:59:03.501620 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:59:03.501583 4821 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49af22e_9041_4f66_a3ff_175ad6d7b342.slice/crio-d2a8b524ca4f295ef4c469ea4eaf324974d9e43c62f9eaa134b81e92c2aa8a2c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49af22e_9041_4f66_a3ff_175ad6d7b342.slice/crio-conmon-d2a8b524ca4f295ef4c469ea4eaf324974d9e43c62f9eaa134b81e92c2aa8a2c.scope\": RecentStats: unable to find data in memory cache]" Oct 29 22:59:03.788802 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.788771 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:59:03.879421 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.876931 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-credential-socket\") pod \"c49af22e-9041-4f66-a3ff-175ad6d7b342\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " Oct 29 22:59:03.879421 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.877087 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-envoy\") pod \"c49af22e-9041-4f66-a3ff-175ad6d7b342\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " Oct 29 22:59:03.879421 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.877159 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-podinfo\") pod \"c49af22e-9041-4f66-a3ff-175ad6d7b342\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " Oct 29 22:59:03.879421 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.877256 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-data\") pod \"c49af22e-9041-4f66-a3ff-175ad6d7b342\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " Oct 29 22:59:03.879421 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.877330 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c49af22e-9041-4f66-a3ff-175ad6d7b342-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49af22e-9041-4f66-a3ff-175ad6d7b342-istiod-ca-cert\") pod \"c49af22e-9041-4f66-a3ff-175ad6d7b342\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " Oct 29 22:59:03.879421 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.877421 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c49af22e-9041-4f66-a3ff-175ad6d7b342-kube-api-access-tkdk7\" (UniqueName: \"kubernetes.io/projected/c49af22e-9041-4f66-a3ff-175ad6d7b342-kube-api-access-tkdk7\") pod \"c49af22e-9041-4f66-a3ff-175ad6d7b342\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " Oct 29 22:59:03.879421 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.877510 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-workload-socket\") pod \"c49af22e-9041-4f66-a3ff-175ad6d7b342\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " Oct 29 22:59:03.879421 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.877584 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-workload-certs\") pod \"c49af22e-9041-4f66-a3ff-175ad6d7b342\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " Oct 29 22:59:03.879421 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.877647 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-token\" (UniqueName: \"kubernetes.io/projected/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-token\") pod \"c49af22e-9041-4f66-a3ff-175ad6d7b342\" (UID: \"c49af22e-9041-4f66-a3ff-175ad6d7b342\") " Oct 29 22:59:03.879421 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.877759 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-credential-socket" pod "c49af22e-9041-4f66-a3ff-175ad6d7b342" (UID: "c49af22e-9041-4f66-a3ff-175ad6d7b342"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:03.879421 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.878602 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:59:03.881539 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.881289 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-token" pod "c49af22e-9041-4f66-a3ff-175ad6d7b342" (UID: "c49af22e-9041-4f66-a3ff-175ad6d7b342"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:03.881539 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.881311 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-podinfo" pod "c49af22e-9041-4f66-a3ff-175ad6d7b342" (UID: "c49af22e-9041-4f66-a3ff-175ad6d7b342"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:59:03.881869 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.881803 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-workload-socket" pod "c49af22e-9041-4f66-a3ff-175ad6d7b342" (UID: "c49af22e-9041-4f66-a3ff-175ad6d7b342"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:03.888523 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.885423 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c49af22e-9041-4f66-a3ff-175ad6d7b342-kube-api-access-tkdk7" pod "c49af22e-9041-4f66-a3ff-175ad6d7b342" (UID: "c49af22e-9041-4f66-a3ff-175ad6d7b342"). InnerVolumeSpecName "kube-api-access-tkdk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:03.888523 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.885542 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c49af22e-9041-4f66-a3ff-175ad6d7b342-istiod-ca-cert" pod "c49af22e-9041-4f66-a3ff-175ad6d7b342" (UID: "c49af22e-9041-4f66-a3ff-175ad6d7b342"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:03.888523 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.885559 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-data" pod "c49af22e-9041-4f66-a3ff-175ad6d7b342" (UID: "c49af22e-9041-4f66-a3ff-175ad6d7b342"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:03.888523 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.885629 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-envoy" pod "c49af22e-9041-4f66-a3ff-175ad6d7b342" (UID: "c49af22e-9041-4f66-a3ff-175ad6d7b342"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:03.888523 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.885716 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-workload-certs" pod "c49af22e-9041-4f66-a3ff-175ad6d7b342" (UID: "c49af22e-9041-4f66-a3ff-175ad6d7b342"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:59:03.979925 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.979888 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:59:03.979925 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.979914 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:59:03.979925 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.979923 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:59:03.979925 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.979935 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49af22e-9041-4f66-a3ff-175ad6d7b342-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:59:03.980291 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.979947 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkdk7\" (UniqueName: \"kubernetes.io/projected/c49af22e-9041-4f66-a3ff-175ad6d7b342-kube-api-access-tkdk7\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:59:03.980291 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.979957 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:59:03.980291 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.979966 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49af22e-9041-4f66-a3ff-175ad6d7b342-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:59:03.980291 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.979974 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49af22e-9041-4f66-a3ff-175ad6d7b342-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:59:03.984987 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.984952 4821 generic.go:356] "Generic (PLEG): container finished" podID="c49af22e-9041-4f66-a3ff-175ad6d7b342" containerID="d2a8b524ca4f295ef4c469ea4eaf324974d9e43c62f9eaa134b81e92c2aa8a2c" exitCode=0 Oct 29 22:59:03.985140 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.985019 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" Oct 29 22:59:03.985140 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.985046 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" event={"ID":"c49af22e-9041-4f66-a3ff-175ad6d7b342","Type":"ContainerDied","Data":"d2a8b524ca4f295ef4c469ea4eaf324974d9e43c62f9eaa134b81e92c2aa8a2c"} Oct 29 22:59:03.985260 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.985144 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b" event={"ID":"c49af22e-9041-4f66-a3ff-175ad6d7b342","Type":"ContainerDied","Data":"eb6b13c88aa1179fcd77043479cef279cc3a0470e57a2ff9cd1d2eed59b22791"} Oct 29 22:59:03.985260 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.985175 4821 scope.go:122] "RemoveContainer" containerID="d2a8b524ca4f295ef4c469ea4eaf324974d9e43c62f9eaa134b81e92c2aa8a2c" Oct 29 22:59:03.997292 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.997276 4821 scope.go:122] "RemoveContainer" containerID="d2a8b524ca4f295ef4c469ea4eaf324974d9e43c62f9eaa134b81e92c2aa8a2c" Oct 29 22:59:03.997566 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:59:03.997548 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2a8b524ca4f295ef4c469ea4eaf324974d9e43c62f9eaa134b81e92c2aa8a2c\": container with ID starting with d2a8b524ca4f295ef4c469ea4eaf324974d9e43c62f9eaa134b81e92c2aa8a2c not found: ID does not exist" containerID="d2a8b524ca4f295ef4c469ea4eaf324974d9e43c62f9eaa134b81e92c2aa8a2c" Oct 29 22:59:03.997614 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:03.997577 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2a8b524ca4f295ef4c469ea4eaf324974d9e43c62f9eaa134b81e92c2aa8a2c"} err="failed to get container status \"d2a8b524ca4f295ef4c469ea4eaf324974d9e43c62f9eaa134b81e92c2aa8a2c\": rpc error: code = NotFound desc = could not find container \"d2a8b524ca4f295ef4c469ea4eaf324974d9e43c62f9eaa134b81e92c2aa8a2c\": container with ID starting with d2a8b524ca4f295ef4c469ea4eaf324974d9e43c62f9eaa134b81e92c2aa8a2c not found: ID does not exist" Oct 29 22:59:04.046310 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:04.046187 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b"] Oct 29 22:59:04.050566 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:04.050538 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e4a049-openshift-default-55bd8fc7f9-mwv4b"] Oct 29 22:59:04.171928 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:04.171893 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c49af22e-9041-4f66-a3ff-175ad6d7b342" path="/var/lib/kubelet/pods/c49af22e-9041-4f66-a3ff-175ad6d7b342/volumes" Oct 29 22:59:10.057210 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:10.057174 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72"] Oct 29 22:59:10.097714 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:10.097678 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86"] Oct 29 22:59:13.065875 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:13.065838 4821 generic.go:356] "Generic (PLEG): container finished" podID="34564236-1496-484d-9fc9-8c35675b3dca" containerID="d01cd83e27d2729b28c647d5632daf26ad1bbbba5cc498bcb284a9e97b9f3c66" exitCode=0 Oct 29 22:59:13.066362 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:13.065907 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" event={"ID":"34564236-1496-484d-9fc9-8c35675b3dca","Type":"ContainerDied","Data":"d01cd83e27d2729b28c647d5632daf26ad1bbbba5cc498bcb284a9e97b9f3c66"} Oct 29 22:59:13.066413 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:13.066386 4821 scope.go:122] "RemoveContainer" containerID="d01cd83e27d2729b28c647d5632daf26ad1bbbba5cc498bcb284a9e97b9f3c66" Oct 29 22:59:13.068274 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:13.068240 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:59:13.068407 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:13.068311 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:59:13.068407 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:13.068343 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 22:59:14.086815 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:14.086784 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" event={"ID":"34564236-1496-484d-9fc9-8c35675b3dca","Type":"ContainerStarted","Data":"1c3afb1f49ba46d69558faca071166b4d79b0d2bf7db86741f435697236920b1"} Oct 29 22:59:14.989121 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:14.989085 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:59:14.990649 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:14.990617 4821 patch_prober.go:29] interesting pod/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:59:14.990803 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:14.990680 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" podUID="34564236-1496-484d-9fc9-8c35675b3dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 29 22:59:15.058066 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:15.058028 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72" podUID="478131d3-996c-4d45-ab02-52cf354b8de9" containerName="payload-processing" containerID="cri-o://416fea6c69d5c8d26d25a6da0430db3d1619f7b1297dec0b84a9d0d062be9750" gracePeriod=25 Oct 29 22:59:15.104110 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:15.104035 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86" podUID="8c859b9a-7e01-4662-8fde-84242e0c387d" containerName="payload-pre-processing" containerID="cri-o://b4bf150559f75a9bc4e4e24d5eba4e2f49caeb1ef004bdfdf222068ae5cdd7c5" gracePeriod=25 Oct 29 22:59:15.491431 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:15.491054 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72" Oct 29 22:59:15.557766 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:15.557719 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86" Oct 29 22:59:15.583003 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:15.582974 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/478131d3-996c-4d45-ab02-52cf354b8de9-kube-api-access-hksw6\" (UniqueName: \"kubernetes.io/projected/478131d3-996c-4d45-ab02-52cf354b8de9-kube-api-access-hksw6\") pod \"478131d3-996c-4d45-ab02-52cf354b8de9\" (UID: \"478131d3-996c-4d45-ab02-52cf354b8de9\") " Oct 29 22:59:15.583450 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:15.583428 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/478131d3-996c-4d45-ab02-52cf354b8de9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478131d3-996c-4d45-ab02-52cf354b8de9-plugins-config-volume\") pod \"478131d3-996c-4d45-ab02-52cf354b8de9\" (UID: \"478131d3-996c-4d45-ab02-52cf354b8de9\") " Oct 29 22:59:15.583881 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:15.583843 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/478131d3-996c-4d45-ab02-52cf354b8de9-plugins-config-volume" pod "478131d3-996c-4d45-ab02-52cf354b8de9" (UID: "478131d3-996c-4d45-ab02-52cf354b8de9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:15.584295 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:15.584275 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478131d3-996c-4d45-ab02-52cf354b8de9-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:59:15.585728 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:15.585704 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/478131d3-996c-4d45-ab02-52cf354b8de9-kube-api-access-hksw6" pod "478131d3-996c-4d45-ab02-52cf354b8de9" (UID: "478131d3-996c-4d45-ab02-52cf354b8de9"). InnerVolumeSpecName "kube-api-access-hksw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:15.649838 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:15.649805 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct"] Oct 29 22:59:15.656632 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:15.656588 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct" podUID="128877c1-c77d-4d28-abc6-f49c24661c4a" containerName="maas-api" containerID="cri-o://d6d481d30083835ef6e4a575fd5379c0b5fcb927882303ed6293f6891dbe6cd4" gracePeriod=30 Oct 29 22:59:15.685879 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:15.685845 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c859b9a-7e01-4662-8fde-84242e0c387d-kube-api-access-krg76\" (UniqueName: \"kubernetes.io/projected/8c859b9a-7e01-4662-8fde-84242e0c387d-kube-api-access-krg76\") pod \"8c859b9a-7e01-4662-8fde-84242e0c387d\" (UID: \"8c859b9a-7e01-4662-8fde-84242e0c387d\") " Oct 29 22:59:15.686036 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:15.685999 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c859b9a-7e01-4662-8fde-84242e0c387d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c859b9a-7e01-4662-8fde-84242e0c387d-plugins-config-volume\") pod \"8c859b9a-7e01-4662-8fde-84242e0c387d\" (UID: \"8c859b9a-7e01-4662-8fde-84242e0c387d\") " Oct 29 22:59:15.687047 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:15.687022 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hksw6\" (UniqueName: \"kubernetes.io/projected/478131d3-996c-4d45-ab02-52cf354b8de9-kube-api-access-hksw6\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:59:15.696362 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:15.696331 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c859b9a-7e01-4662-8fde-84242e0c387d-kube-api-access-krg76" pod "8c859b9a-7e01-4662-8fde-84242e0c387d" (UID: "8c859b9a-7e01-4662-8fde-84242e0c387d"). InnerVolumeSpecName "kube-api-access-krg76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:15.696553 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:15.696531 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c859b9a-7e01-4662-8fde-84242e0c387d-plugins-config-volume" pod "8c859b9a-7e01-4662-8fde-84242e0c387d" (UID: "8c859b9a-7e01-4662-8fde-84242e0c387d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:15.788984 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:15.788948 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c859b9a-7e01-4662-8fde-84242e0c387d-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:59:15.788984 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:15.788987 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krg76\" (UniqueName: \"kubernetes.io/projected/8c859b9a-7e01-4662-8fde-84242e0c387d-kube-api-access-krg76\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:59:16.021225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.021031 4821 patch_prober.go:29] interesting pod/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:59:16.021225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.021089 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" podUID="34564236-1496-484d-9fc9-8c35675b3dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 29 22:59:16.489505 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.488388 4821 generic.go:356] "Generic (PLEG): container finished" podID="128877c1-c77d-4d28-abc6-f49c24661c4a" containerID="d6d481d30083835ef6e4a575fd5379c0b5fcb927882303ed6293f6891dbe6cd4" exitCode=0 Oct 29 22:59:16.491657 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.491052 4821 generic.go:356] "Generic (PLEG): container finished" podID="8c859b9a-7e01-4662-8fde-84242e0c387d" containerID="b4bf150559f75a9bc4e4e24d5eba4e2f49caeb1ef004bdfdf222068ae5cdd7c5" exitCode=0 Oct 29 22:59:16.495512 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.493342 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86" Oct 29 22:59:16.504349 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.504304 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct" event={"ID":"128877c1-c77d-4d28-abc6-f49c24661c4a","Type":"ContainerDied","Data":"d6d481d30083835ef6e4a575fd5379c0b5fcb927882303ed6293f6891dbe6cd4"} Oct 29 22:59:16.518394 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.514123 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:59:16.518394 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.514160 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86" event={"ID":"8c859b9a-7e01-4662-8fde-84242e0c387d","Type":"ContainerDied","Data":"b4bf150559f75a9bc4e4e24d5eba4e2f49caeb1ef004bdfdf222068ae5cdd7c5"} Oct 29 22:59:16.518394 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.514191 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86" event={"ID":"8c859b9a-7e01-4662-8fde-84242e0c387d","Type":"ContainerDied","Data":"5a37868f2acadc031c6932af5c4e34ce77331aafc54d66c32b73455cbea08ef0"} Oct 29 22:59:16.518394 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.514264 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72" event={"ID":"478131d3-996c-4d45-ab02-52cf354b8de9","Type":"ContainerDied","Data":"416fea6c69d5c8d26d25a6da0430db3d1619f7b1297dec0b84a9d0d062be9750"} Oct 29 22:59:16.518394 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.514297 4821 scope.go:122] "RemoveContainer" containerID="b4bf150559f75a9bc4e4e24d5eba4e2f49caeb1ef004bdfdf222068ae5cdd7c5" Oct 29 22:59:16.518394 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.504338 4821 generic.go:356] "Generic (PLEG): container finished" podID="478131d3-996c-4d45-ab02-52cf354b8de9" containerID="416fea6c69d5c8d26d25a6da0430db3d1619f7b1297dec0b84a9d0d062be9750" exitCode=0 Oct 29 22:59:16.533368 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.504390 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72" Oct 29 22:59:16.542488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.542458 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct" Oct 29 22:59:16.544817 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.544790 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72" event={"ID":"478131d3-996c-4d45-ab02-52cf354b8de9","Type":"ContainerDied","Data":"ee04797d476f7091093d89217ef16408daa8d6c7dcc507dc69426514d3cff173"} Oct 29 22:59:16.561826 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.561804 4821 scope.go:122] "RemoveContainer" containerID="b4bf150559f75a9bc4e4e24d5eba4e2f49caeb1ef004bdfdf222068ae5cdd7c5" Oct 29 22:59:16.562276 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:59:16.562102 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4bf150559f75a9bc4e4e24d5eba4e2f49caeb1ef004bdfdf222068ae5cdd7c5\": container with ID starting with b4bf150559f75a9bc4e4e24d5eba4e2f49caeb1ef004bdfdf222068ae5cdd7c5 not found: ID does not exist" containerID="b4bf150559f75a9bc4e4e24d5eba4e2f49caeb1ef004bdfdf222068ae5cdd7c5" Oct 29 22:59:16.562276 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.562134 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4bf150559f75a9bc4e4e24d5eba4e2f49caeb1ef004bdfdf222068ae5cdd7c5"} err="failed to get container status \"b4bf150559f75a9bc4e4e24d5eba4e2f49caeb1ef004bdfdf222068ae5cdd7c5\": rpc error: code = NotFound desc = could not find container \"b4bf150559f75a9bc4e4e24d5eba4e2f49caeb1ef004bdfdf222068ae5cdd7c5\": container with ID starting with b4bf150559f75a9bc4e4e24d5eba4e2f49caeb1ef004bdfdf222068ae5cdd7c5 not found: ID does not exist" Oct 29 22:59:16.562276 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.562152 4821 scope.go:122] "RemoveContainer" containerID="416fea6c69d5c8d26d25a6da0430db3d1619f7b1297dec0b84a9d0d062be9750" Oct 29 22:59:16.574168 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.574149 4821 scope.go:122] "RemoveContainer" containerID="416fea6c69d5c8d26d25a6da0430db3d1619f7b1297dec0b84a9d0d062be9750" Oct 29 22:59:16.574597 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:59:16.574561 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"416fea6c69d5c8d26d25a6da0430db3d1619f7b1297dec0b84a9d0d062be9750\": container with ID starting with 416fea6c69d5c8d26d25a6da0430db3d1619f7b1297dec0b84a9d0d062be9750 not found: ID does not exist" containerID="416fea6c69d5c8d26d25a6da0430db3d1619f7b1297dec0b84a9d0d062be9750" Oct 29 22:59:16.574696 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.574599 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"416fea6c69d5c8d26d25a6da0430db3d1619f7b1297dec0b84a9d0d062be9750"} err="failed to get container status \"416fea6c69d5c8d26d25a6da0430db3d1619f7b1297dec0b84a9d0d062be9750\": rpc error: code = NotFound desc = could not find container \"416fea6c69d5c8d26d25a6da0430db3d1619f7b1297dec0b84a9d0d062be9750\": container with ID starting with 416fea6c69d5c8d26d25a6da0430db3d1619f7b1297dec0b84a9d0d062be9750 not found: ID does not exist" Oct 29 22:59:16.644224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.642821 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/128877c1-c77d-4d28-abc6-f49c24661c4a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/128877c1-c77d-4d28-abc6-f49c24661c4a-metrics-certs\") pod \"128877c1-c77d-4d28-abc6-f49c24661c4a\" (UID: \"128877c1-c77d-4d28-abc6-f49c24661c4a\") " Oct 29 22:59:16.644224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.642965 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/128877c1-c77d-4d28-abc6-f49c24661c4a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/128877c1-c77d-4d28-abc6-f49c24661c4a-openshift-service-ca\") pod \"128877c1-c77d-4d28-abc6-f49c24661c4a\" (UID: \"128877c1-c77d-4d28-abc6-f49c24661c4a\") " Oct 29 22:59:16.644224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.643121 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/128877c1-c77d-4d28-abc6-f49c24661c4a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/128877c1-c77d-4d28-abc6-f49c24661c4a-maas-api-tls\") pod \"128877c1-c77d-4d28-abc6-f49c24661c4a\" (UID: \"128877c1-c77d-4d28-abc6-f49c24661c4a\") " Oct 29 22:59:16.644224 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.643168 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/128877c1-c77d-4d28-abc6-f49c24661c4a-kube-api-access-jjrz7\" (UniqueName: \"kubernetes.io/projected/128877c1-c77d-4d28-abc6-f49c24661c4a-kube-api-access-jjrz7\") pod \"128877c1-c77d-4d28-abc6-f49c24661c4a\" (UID: \"128877c1-c77d-4d28-abc6-f49c24661c4a\") " Oct 29 22:59:16.653226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.649660 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/128877c1-c77d-4d28-abc6-f49c24661c4a-openshift-service-ca" pod "128877c1-c77d-4d28-abc6-f49c24661c4a" (UID: "128877c1-c77d-4d28-abc6-f49c24661c4a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:16.666594 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.666530 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/128877c1-c77d-4d28-abc6-f49c24661c4a-kube-api-access-jjrz7" pod "128877c1-c77d-4d28-abc6-f49c24661c4a" (UID: "128877c1-c77d-4d28-abc6-f49c24661c4a"). InnerVolumeSpecName "kube-api-access-jjrz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:16.668061 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.668034 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/128877c1-c77d-4d28-abc6-f49c24661c4a-maas-api-tls" pod "128877c1-c77d-4d28-abc6-f49c24661c4a" (UID: "128877c1-c77d-4d28-abc6-f49c24661c4a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:59:16.683779 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.683276 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/128877c1-c77d-4d28-abc6-f49c24661c4a-metrics-certs" pod "128877c1-c77d-4d28-abc6-f49c24661c4a" (UID: "128877c1-c77d-4d28-abc6-f49c24661c4a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:59:16.693522 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.693486 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72"] Oct 29 22:59:16.697522 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.697495 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-27d16a-7f578d6fd-k5n72"] Oct 29 22:59:16.752343 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.747734 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/128877c1-c77d-4d28-abc6-f49c24661c4a-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:59:16.752343 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.747774 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjrz7\" (UniqueName: \"kubernetes.io/projected/128877c1-c77d-4d28-abc6-f49c24661c4a-kube-api-access-jjrz7\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:59:16.752343 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.747789 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/128877c1-c77d-4d28-abc6-f49c24661c4a-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:59:16.752343 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.747803 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/128877c1-c77d-4d28-abc6-f49c24661c4a-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:59:16.757161 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.757130 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86"] Oct 29 22:59:16.761670 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.761650 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-27d16a-7f69bc559-bgq86"] Oct 29 22:59:16.991415 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.991369 4821 patch_prober.go:29] interesting pod/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:59:16.991635 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:16.991499 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" podUID="34564236-1496-484d-9fc9-8c35675b3dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 29 22:59:17.549720 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:17.549695 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct" Oct 29 22:59:17.550399 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:17.549688 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct" event={"ID":"128877c1-c77d-4d28-abc6-f49c24661c4a","Type":"ContainerDied","Data":"a08faf52c1c010b2b27fd6804acd1dac76adbfaa0c5106b2807cd2780356ef33"} Oct 29 22:59:17.550502 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:17.550447 4821 scope.go:122] "RemoveContainer" containerID="d6d481d30083835ef6e4a575fd5379c0b5fcb927882303ed6293f6891dbe6cd4" Oct 29 22:59:17.603823 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:17.603792 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct"] Oct 29 22:59:17.634226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:17.618265 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a37a29-6bf956ccd4-twjct"] Oct 29 22:59:17.989747 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:17.989706 4821 patch_prober.go:29] interesting pod/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:59:17.989948 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:17.989774 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" podUID="34564236-1496-484d-9fc9-8c35675b3dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 29 22:59:18.172054 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:18.172022 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="128877c1-c77d-4d28-abc6-f49c24661c4a" path="/var/lib/kubelet/pods/128877c1-c77d-4d28-abc6-f49c24661c4a/volumes" Oct 29 22:59:18.172521 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:18.172507 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="478131d3-996c-4d45-ab02-52cf354b8de9" path="/var/lib/kubelet/pods/478131d3-996c-4d45-ab02-52cf354b8de9/volumes" Oct 29 22:59:18.172931 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:18.172915 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c859b9a-7e01-4662-8fde-84242e0c387d" path="/var/lib/kubelet/pods/8c859b9a-7e01-4662-8fde-84242e0c387d/volumes" Oct 29 22:59:18.989481 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:18.989181 4821 patch_prober.go:29] interesting pod/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:59:18.989481 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:18.989265 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" podUID="34564236-1496-484d-9fc9-8c35675b3dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 29 22:59:19.991381 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:19.991346 4821 patch_prober.go:29] interesting pod/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 29 22:59:19.992071 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:19.992029 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" podUID="34564236-1496-484d-9fc9-8c35675b3dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 29 22:59:20.994698 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:20.994666 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:59:20.995776 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:20.995750 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 22:59:23.889035 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:59:23.889000 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:59:25.274300 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:25.274260 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb"] Oct 29 22:59:30.274791 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:30.274746 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb" podUID="6a8afe32-2de7-4a50-aced-ce5bf49c2db6" containerName="payload-processing" containerID="cri-o://32176ce337ac01678e66ccd6e38be1ae7321dd5d540668443bde83a5f0723a6a" gracePeriod=25 Oct 29 22:59:30.563395 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:30.563362 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb" Oct 29 22:59:30.713927 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:30.713892 4821 generic.go:356] "Generic (PLEG): container finished" podID="6a8afe32-2de7-4a50-aced-ce5bf49c2db6" containerID="32176ce337ac01678e66ccd6e38be1ae7321dd5d540668443bde83a5f0723a6a" exitCode=0 Oct 29 22:59:30.714107 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:30.713936 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb" Oct 29 22:59:30.714107 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:30.713966 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb" event={"ID":"6a8afe32-2de7-4a50-aced-ce5bf49c2db6","Type":"ContainerDied","Data":"32176ce337ac01678e66ccd6e38be1ae7321dd5d540668443bde83a5f0723a6a"} Oct 29 22:59:30.714107 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:30.714021 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb" event={"ID":"6a8afe32-2de7-4a50-aced-ce5bf49c2db6","Type":"ContainerDied","Data":"58ec7ce333236d4d0a94555e79db66e7e5a6aab2a7b662511790908fd1b17ba6"} Oct 29 22:59:30.714107 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:30.714052 4821 scope.go:122] "RemoveContainer" containerID="32176ce337ac01678e66ccd6e38be1ae7321dd5d540668443bde83a5f0723a6a" Oct 29 22:59:30.726755 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:30.726732 4821 scope.go:122] "RemoveContainer" containerID="32176ce337ac01678e66ccd6e38be1ae7321dd5d540668443bde83a5f0723a6a" Oct 29 22:59:30.727133 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:59:30.727111 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32176ce337ac01678e66ccd6e38be1ae7321dd5d540668443bde83a5f0723a6a\": container with ID starting with 32176ce337ac01678e66ccd6e38be1ae7321dd5d540668443bde83a5f0723a6a not found: ID does not exist" containerID="32176ce337ac01678e66ccd6e38be1ae7321dd5d540668443bde83a5f0723a6a" Oct 29 22:59:30.727228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:30.727142 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32176ce337ac01678e66ccd6e38be1ae7321dd5d540668443bde83a5f0723a6a"} err="failed to get container status \"32176ce337ac01678e66ccd6e38be1ae7321dd5d540668443bde83a5f0723a6a\": rpc error: code = NotFound desc = could not find container \"32176ce337ac01678e66ccd6e38be1ae7321dd5d540668443bde83a5f0723a6a\": container with ID starting with 32176ce337ac01678e66ccd6e38be1ae7321dd5d540668443bde83a5f0723a6a not found: ID does not exist" Oct 29 22:59:30.751893 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:30.751858 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a8afe32-2de7-4a50-aced-ce5bf49c2db6-kube-api-access-jdv2d\" (UniqueName: \"kubernetes.io/projected/6a8afe32-2de7-4a50-aced-ce5bf49c2db6-kube-api-access-jdv2d\") pod \"6a8afe32-2de7-4a50-aced-ce5bf49c2db6\" (UID: \"6a8afe32-2de7-4a50-aced-ce5bf49c2db6\") " Oct 29 22:59:30.752109 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:30.751936 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a8afe32-2de7-4a50-aced-ce5bf49c2db6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a8afe32-2de7-4a50-aced-ce5bf49c2db6-plugins-config-volume\") pod \"6a8afe32-2de7-4a50-aced-ce5bf49c2db6\" (UID: \"6a8afe32-2de7-4a50-aced-ce5bf49c2db6\") " Oct 29 22:59:30.752488 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:30.752452 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a8afe32-2de7-4a50-aced-ce5bf49c2db6-plugins-config-volume" pod "6a8afe32-2de7-4a50-aced-ce5bf49c2db6" (UID: "6a8afe32-2de7-4a50-aced-ce5bf49c2db6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:30.752716 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:30.752697 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a8afe32-2de7-4a50-aced-ce5bf49c2db6-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:59:30.754672 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:30.754643 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a8afe32-2de7-4a50-aced-ce5bf49c2db6-kube-api-access-jdv2d" pod "6a8afe32-2de7-4a50-aced-ce5bf49c2db6" (UID: "6a8afe32-2de7-4a50-aced-ce5bf49c2db6"). InnerVolumeSpecName "kube-api-access-jdv2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:30.854187 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:30.854103 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdv2d\" (UniqueName: \"kubernetes.io/projected/6a8afe32-2de7-4a50-aced-ce5bf49c2db6-kube-api-access-jdv2d\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:59:31.061803 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:31.061769 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb"] Oct 29 22:59:31.072846 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:31.072821 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd525f-57f9569b64-zw4hb"] Oct 29 22:59:32.171275 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:32.171235 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a8afe32-2de7-4a50-aced-ce5bf49c2db6" path="/var/lib/kubelet/pods/6a8afe32-2de7-4a50-aced-ce5bf49c2db6/volumes" Oct 29 22:59:44.353167 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.353085 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm"] Oct 29 22:59:44.353806 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.353782 4821 cpu_manager.go:414] "containerMap: removing container" podUID="6a8afe32-2de7-4a50-aced-ce5bf49c2db6" containerName="payload-processing" Oct 29 22:59:44.353924 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.353843 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a8afe32-2de7-4a50-aced-ce5bf49c2db6" containerName="payload-processing" Oct 29 22:59:44.353924 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.353883 4821 cpu_manager.go:414] "containerMap: removing container" podUID="8c859b9a-7e01-4662-8fde-84242e0c387d" containerName="payload-pre-processing" Oct 29 22:59:44.353924 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.353891 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c859b9a-7e01-4662-8fde-84242e0c387d" containerName="payload-pre-processing" Oct 29 22:59:44.353924 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.353912 4821 cpu_manager.go:414] "containerMap: removing container" podUID="eea051e6-e43a-473a-9c98-1a018a8e3d27" containerName="payload-processing" Oct 29 22:59:44.353924 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.353919 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea051e6-e43a-473a-9c98-1a018a8e3d27" containerName="payload-processing" Oct 29 22:59:44.354139 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.353960 4821 cpu_manager.go:414] "containerMap: removing container" podUID="c49af22e-9041-4f66-a3ff-175ad6d7b342" containerName="istio-proxy" Oct 29 22:59:44.354139 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.353968 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c49af22e-9041-4f66-a3ff-175ad6d7b342" containerName="istio-proxy" Oct 29 22:59:44.354139 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.353977 4821 cpu_manager.go:414] "containerMap: removing container" podUID="478131d3-996c-4d45-ab02-52cf354b8de9" containerName="payload-processing" Oct 29 22:59:44.354139 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.353983 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="478131d3-996c-4d45-ab02-52cf354b8de9" containerName="payload-processing" Oct 29 22:59:44.354139 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.354019 4821 cpu_manager.go:414] "containerMap: removing container" podUID="128877c1-c77d-4d28-abc6-f49c24661c4a" containerName="maas-api" Oct 29 22:59:44.354139 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.354025 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="128877c1-c77d-4d28-abc6-f49c24661c4a" containerName="maas-api" Oct 29 22:59:44.354139 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.354037 4821 cpu_manager.go:414] "containerMap: removing container" podUID="799abc76-e641-4d92-b48f-d213c02c50ec" containerName="payload-pre-processing" Oct 29 22:59:44.354139 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.354045 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="799abc76-e641-4d92-b48f-d213c02c50ec" containerName="payload-pre-processing" Oct 29 22:59:44.354764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.354265 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a8afe32-2de7-4a50-aced-ce5bf49c2db6" containerName="payload-processing" Oct 29 22:59:44.354764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.354285 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c859b9a-7e01-4662-8fde-84242e0c387d" containerName="payload-pre-processing" Oct 29 22:59:44.354764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.354296 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea051e6-e43a-473a-9c98-1a018a8e3d27" containerName="payload-processing" Oct 29 22:59:44.354764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.354308 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="c49af22e-9041-4f66-a3ff-175ad6d7b342" containerName="istio-proxy" Oct 29 22:59:44.354764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.354315 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="478131d3-996c-4d45-ab02-52cf354b8de9" containerName="payload-processing" Oct 29 22:59:44.354764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.354329 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="128877c1-c77d-4d28-abc6-f49c24661c4a" containerName="maas-api" Oct 29 22:59:44.354764 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.354338 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="799abc76-e641-4d92-b48f-d213c02c50ec" containerName="payload-pre-processing" Oct 29 22:59:44.358131 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.358114 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" Oct 29 22:59:44.379187 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.377525 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-49dbb5-serving-cert\"" Oct 29 22:59:44.388130 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.388101 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm"] Oct 29 22:59:44.407066 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.407029 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03949775-cc5e-4c51-8179-f53230828245-maas-api-tls\") pod \"maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm\" (UID: \"03949775-cc5e-4c51-8179-f53230828245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" Oct 29 22:59:44.407256 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.407100 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9x8z\" (UniqueName: \"kubernetes.io/projected/03949775-cc5e-4c51-8179-f53230828245-kube-api-access-m9x8z\") pod \"maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm\" (UID: \"03949775-cc5e-4c51-8179-f53230828245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" Oct 29 22:59:44.407256 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.407145 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03949775-cc5e-4c51-8179-f53230828245-metrics-certs\") pod \"maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm\" (UID: \"03949775-cc5e-4c51-8179-f53230828245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" Oct 29 22:59:44.407379 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.407276 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03949775-cc5e-4c51-8179-f53230828245-openshift-service-ca\") pod \"maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm\" (UID: \"03949775-cc5e-4c51-8179-f53230828245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" Oct 29 22:59:44.509455 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.509423 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9x8z\" (UniqueName: \"kubernetes.io/projected/03949775-cc5e-4c51-8179-f53230828245-kube-api-access-m9x8z\") pod \"maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm\" (UID: \"03949775-cc5e-4c51-8179-f53230828245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" Oct 29 22:59:44.509455 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.509473 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03949775-cc5e-4c51-8179-f53230828245-metrics-certs\") pod \"maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm\" (UID: \"03949775-cc5e-4c51-8179-f53230828245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" Oct 29 22:59:44.509702 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.509556 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03949775-cc5e-4c51-8179-f53230828245-openshift-service-ca\") pod \"maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm\" (UID: \"03949775-cc5e-4c51-8179-f53230828245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" Oct 29 22:59:44.509774 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.509760 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03949775-cc5e-4c51-8179-f53230828245-maas-api-tls\") pod \"maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm\" (UID: \"03949775-cc5e-4c51-8179-f53230828245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" Oct 29 22:59:44.509832 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:59:44.509816 4821 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-serving-cert: secret "maas-api-e2e-api-a-49dbb5-serving-cert" not found Oct 29 22:59:44.509916 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:59:44.509888 4821 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03949775-cc5e-4c51-8179-f53230828245-maas-api-tls podName:03949775-cc5e-4c51-8179-f53230828245 nodeName:}" failed. No retries permitted until 2026-10-29 22:59:45.009872896 +0000 UTC m=+1038.950420783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/03949775-cc5e-4c51-8179-f53230828245-maas-api-tls") pod "maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" (UID: "03949775-cc5e-4c51-8179-f53230828245") : secret "maas-api-e2e-api-a-49dbb5-serving-cert" not found Oct 29 22:59:44.510397 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.510370 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03949775-cc5e-4c51-8179-f53230828245-openshift-service-ca\") pod \"maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm\" (UID: \"03949775-cc5e-4c51-8179-f53230828245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" Oct 29 22:59:44.512262 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.512241 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03949775-cc5e-4c51-8179-f53230828245-metrics-certs\") pod \"maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm\" (UID: \"03949775-cc5e-4c51-8179-f53230828245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" Oct 29 22:59:44.559710 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:44.559675 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9x8z\" (UniqueName: \"kubernetes.io/projected/03949775-cc5e-4c51-8179-f53230828245-kube-api-access-m9x8z\") pod \"maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm\" (UID: \"03949775-cc5e-4c51-8179-f53230828245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" Oct 29 22:59:45.076327 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:45.076286 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03949775-cc5e-4c51-8179-f53230828245-maas-api-tls\") pod \"maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm\" (UID: \"03949775-cc5e-4c51-8179-f53230828245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" Oct 29 22:59:45.090226 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:45.082065 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03949775-cc5e-4c51-8179-f53230828245-maas-api-tls\") pod \"maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm\" (UID: \"03949775-cc5e-4c51-8179-f53230828245\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" Oct 29 22:59:45.282308 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:45.282272 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" Oct 29 22:59:45.614306 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:45.609878 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm"] Oct 29 22:59:45.845711 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:45.845671 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" event={"ID":"03949775-cc5e-4c51-8179-f53230828245","Type":"ContainerStarted","Data":"5bbc9941339953533a98a2dcaca9a24bd52f3fe8e1fc9365a6f1c504a3b4f4ce"} Oct 29 22:59:46.855682 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:46.855628 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" event={"ID":"03949775-cc5e-4c51-8179-f53230828245","Type":"ContainerStarted","Data":"04b46a3472fa5f1a74e1a7a158c3a7e0e288222bbaa7952f0e874e3772ce6947"} Oct 29 22:59:46.856188 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:46.855725 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" Oct 29 22:59:46.902076 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:46.902022 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" podStartSLOduration=2.461199703 podStartE2EDuration="2.902006963s" totalImagesPullingTime="440.80726ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:46.896981482 +0000 UTC m=+1040.837529405" watchObservedRunningTime="2026-10-29 22:59:46.902006963 +0000 UTC m=+1040.842554925" Oct 29 22:59:48.375742 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:48.375711 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs"] Oct 29 22:59:48.380235 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:48.380194 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs" Oct 29 22:59:48.403528 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:48.403300 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs"] Oct 29 22:59:48.409555 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:48.409529 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-13aa41-dockercfg-s4w8p\"" Oct 29 22:59:48.409712 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:48.409622 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-13aa41\"" Oct 29 22:59:48.551636 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:48.544257 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t"] Oct 29 22:59:48.555316 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:48.553563 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t" Oct 29 22:59:48.584225 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:48.560350 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrl47\" (UniqueName: \"kubernetes.io/projected/24b8973e-4340-4632-8992-c7a80d40af90-kube-api-access-nrl47\") pod \"payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs\" (UID: \"24b8973e-4340-4632-8992-c7a80d40af90\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs" Oct 29 22:59:48.584593 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:48.584568 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b8973e-4340-4632-8992-c7a80d40af90-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs\" (UID: \"24b8973e-4340-4632-8992-c7a80d40af90\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs" Oct 29 22:59:48.695384 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:48.688760 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t"] Oct 29 22:59:48.695384 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:48.690800 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7012b82-2cf3-4b96-8646-2434d6aab454-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t\" (UID: \"d7012b82-2cf3-4b96-8646-2434d6aab454\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t" Oct 29 22:59:48.695384 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:48.690933 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrl47\" (UniqueName: \"kubernetes.io/projected/24b8973e-4340-4632-8992-c7a80d40af90-kube-api-access-nrl47\") pod \"payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs\" (UID: \"24b8973e-4340-4632-8992-c7a80d40af90\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs" Oct 29 22:59:48.695384 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:48.691012 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b8973e-4340-4632-8992-c7a80d40af90-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs\" (UID: \"24b8973e-4340-4632-8992-c7a80d40af90\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs" Oct 29 22:59:48.695384 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:48.691075 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p5l6\" (UniqueName: \"kubernetes.io/projected/d7012b82-2cf3-4b96-8646-2434d6aab454-kube-api-access-8p5l6\") pod \"payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t\" (UID: \"d7012b82-2cf3-4b96-8646-2434d6aab454\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t" Oct 29 22:59:48.695384 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:48.692463 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b8973e-4340-4632-8992-c7a80d40af90-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs\" (UID: \"24b8973e-4340-4632-8992-c7a80d40af90\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs" Oct 29 22:59:48.718341 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:48.718234 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrl47\" (UniqueName: \"kubernetes.io/projected/24b8973e-4340-4632-8992-c7a80d40af90-kube-api-access-nrl47\") pod \"payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs\" (UID: \"24b8973e-4340-4632-8992-c7a80d40af90\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs" Oct 29 22:59:48.792044 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:48.792013 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p5l6\" (UniqueName: \"kubernetes.io/projected/d7012b82-2cf3-4b96-8646-2434d6aab454-kube-api-access-8p5l6\") pod \"payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t\" (UID: \"d7012b82-2cf3-4b96-8646-2434d6aab454\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t" Oct 29 22:59:48.792249 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:48.792107 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7012b82-2cf3-4b96-8646-2434d6aab454-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t\" (UID: \"d7012b82-2cf3-4b96-8646-2434d6aab454\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t" Oct 29 22:59:48.792811 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:48.792792 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7012b82-2cf3-4b96-8646-2434d6aab454-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t\" (UID: \"d7012b82-2cf3-4b96-8646-2434d6aab454\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t" Oct 29 22:59:48.815122 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:48.815090 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p5l6\" (UniqueName: \"kubernetes.io/projected/d7012b82-2cf3-4b96-8646-2434d6aab454-kube-api-access-8p5l6\") pod \"payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t\" (UID: \"d7012b82-2cf3-4b96-8646-2434d6aab454\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t" Oct 29 22:59:48.993761 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:48.993675 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t" Oct 29 22:59:48.995387 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:48.995356 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs" Oct 29 22:59:49.356925 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:59:49.356878 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b8973e_4340_4632_8992_c7a80d40af90.slice/crio-5f2e8ee58753c0a3fa282310499fa7c67b5105561ee7d38dd63e245adafe139b WatchSource:0}: Error finding container 5f2e8ee58753c0a3fa282310499fa7c67b5105561ee7d38dd63e245adafe139b: Status 404 returned error can't find the container with id 5f2e8ee58753c0a3fa282310499fa7c67b5105561ee7d38dd63e245adafe139b Oct 29 22:59:49.357256 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:49.357235 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs"] Oct 29 22:59:49.357949 ip-10-0-130-252 kubenswrapper[4821]: W1029 22:59:49.357910 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7012b82_2cf3_4b96_8646_2434d6aab454.slice/crio-e61936f650716ebd25cb47296dc08144ffce53b564f1853a3505454dbea3e0df WatchSource:0}: Error finding container e61936f650716ebd25cb47296dc08144ffce53b564f1853a3505454dbea3e0df: Status 404 returned error can't find the container with id e61936f650716ebd25cb47296dc08144ffce53b564f1853a3505454dbea3e0df Oct 29 22:59:49.369655 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:49.369619 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t"] Oct 29 22:59:49.882144 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:49.882099 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs" event={"ID":"24b8973e-4340-4632-8992-c7a80d40af90","Type":"ContainerStarted","Data":"a420252aafc13e35e23bdaa5fbb7369eab544dbecba138d52fced8929d03bd5e"} Oct 29 22:59:49.882688 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:49.882171 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs" event={"ID":"24b8973e-4340-4632-8992-c7a80d40af90","Type":"ContainerStarted","Data":"5f2e8ee58753c0a3fa282310499fa7c67b5105561ee7d38dd63e245adafe139b"} Oct 29 22:59:49.882688 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:49.882345 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs" Oct 29 22:59:49.883919 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:49.883894 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t" event={"ID":"d7012b82-2cf3-4b96-8646-2434d6aab454","Type":"ContainerStarted","Data":"3fefbcc751485a109a14cf2b908053938ad195e2077b87b94020e5b15ea58763"} Oct 29 22:59:49.884046 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:49.883948 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t" Oct 29 22:59:49.884046 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:49.883973 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t" event={"ID":"d7012b82-2cf3-4b96-8646-2434d6aab454","Type":"ContainerStarted","Data":"e61936f650716ebd25cb47296dc08144ffce53b564f1853a3505454dbea3e0df"} Oct 29 22:59:49.924668 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:49.924623 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs" podStartSLOduration=1.924610537 podStartE2EDuration="1.924610537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:49.920632445 +0000 UTC m=+1043.861180355" watchObservedRunningTime="2026-10-29 22:59:49.924610537 +0000 UTC m=+1043.865158517" Oct 29 22:59:49.960885 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:49.960836 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t" podStartSLOduration=1.9608224170000002 podStartE2EDuration="1.960822417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:49.959013891 +0000 UTC m=+1043.899561802" watchObservedRunningTime="2026-10-29 22:59:49.960822417 +0000 UTC m=+1043.901370327" Oct 29 22:59:52.868792 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:52.868763 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" Oct 29 22:59:53.889034 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:59:53.888997 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 22:59:54.115629 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:54.115594 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t"] Oct 29 22:59:54.115895 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:54.115870 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" podUID="d5a6b1fc-4fb1-41ae-99e2-359d9274c11d" containerName="maas-api" containerID="cri-o://ed35dc0a7c7ca1c3aa28e6243f0cd0040166ee9d369e5abe2ac1c610ed10be8b" gracePeriod=30 Oct 29 22:59:54.666288 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:54.661831 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" Oct 29 22:59:54.700228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:54.699820 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-metrics-certs\") pod \"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d\" (UID: \"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d\") " Oct 29 22:59:54.700228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:54.700031 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-kube-api-access-xg97d\" (UniqueName: \"kubernetes.io/projected/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-kube-api-access-xg97d\") pod \"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d\" (UID: \"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d\") " Oct 29 22:59:54.700228 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:54.700112 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-maas-api-tls\") pod \"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d\" (UID: \"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d\") " Oct 29 22:59:54.700553 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:54.700273 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-openshift-service-ca\") pod \"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d\" (UID: \"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d\") " Oct 29 22:59:54.714227 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:54.712577 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-openshift-service-ca" pod "d5a6b1fc-4fb1-41ae-99e2-359d9274c11d" (UID: "d5a6b1fc-4fb1-41ae-99e2-359d9274c11d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:54.745850 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:54.743302 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-metrics-certs" pod "d5a6b1fc-4fb1-41ae-99e2-359d9274c11d" (UID: "d5a6b1fc-4fb1-41ae-99e2-359d9274c11d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:59:54.753626 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:54.748918 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-maas-api-tls" pod "d5a6b1fc-4fb1-41ae-99e2-359d9274c11d" (UID: "d5a6b1fc-4fb1-41ae-99e2-359d9274c11d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:59:54.763649 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:54.763547 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-kube-api-access-xg97d" pod "d5a6b1fc-4fb1-41ae-99e2-359d9274c11d" (UID: "d5a6b1fc-4fb1-41ae-99e2-359d9274c11d"). InnerVolumeSpecName "kube-api-access-xg97d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:54.807987 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:54.807901 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:59:54.808291 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:54.807994 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:59:54.808291 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:54.808011 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg97d\" (UniqueName: \"kubernetes.io/projected/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-kube-api-access-xg97d\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:59:54.808291 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:54.808025 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 22:59:54.933072 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:54.933042 4821 generic.go:356] "Generic (PLEG): container finished" podID="d5a6b1fc-4fb1-41ae-99e2-359d9274c11d" containerID="ed35dc0a7c7ca1c3aa28e6243f0cd0040166ee9d369e5abe2ac1c610ed10be8b" exitCode=0 Oct 29 22:59:54.933600 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:54.933107 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" Oct 29 22:59:54.933600 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:54.933135 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" event={"ID":"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d","Type":"ContainerDied","Data":"ed35dc0a7c7ca1c3aa28e6243f0cd0040166ee9d369e5abe2ac1c610ed10be8b"} Oct 29 22:59:54.933600 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:54.933191 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" event={"ID":"d5a6b1fc-4fb1-41ae-99e2-359d9274c11d","Type":"ContainerDied","Data":"b539c5c98d7ba3a3836b68a4ec357ac53042d204bd2d8d3a875325dab5176b7c"} Oct 29 22:59:54.933600 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:54.933233 4821 scope.go:122] "RemoveContainer" containerID="ed35dc0a7c7ca1c3aa28e6243f0cd0040166ee9d369e5abe2ac1c610ed10be8b" Oct 29 22:59:54.946221 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:54.946185 4821 scope.go:122] "RemoveContainer" containerID="ed35dc0a7c7ca1c3aa28e6243f0cd0040166ee9d369e5abe2ac1c610ed10be8b" Oct 29 22:59:54.946579 ip-10-0-130-252 kubenswrapper[4821]: E1029 22:59:54.946556 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed35dc0a7c7ca1c3aa28e6243f0cd0040166ee9d369e5abe2ac1c610ed10be8b\": container with ID starting with ed35dc0a7c7ca1c3aa28e6243f0cd0040166ee9d369e5abe2ac1c610ed10be8b not found: ID does not exist" containerID="ed35dc0a7c7ca1c3aa28e6243f0cd0040166ee9d369e5abe2ac1c610ed10be8b" Oct 29 22:59:54.946624 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:54.946592 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed35dc0a7c7ca1c3aa28e6243f0cd0040166ee9d369e5abe2ac1c610ed10be8b"} err="failed to get container status \"ed35dc0a7c7ca1c3aa28e6243f0cd0040166ee9d369e5abe2ac1c610ed10be8b\": rpc error: code = NotFound desc = could not find container \"ed35dc0a7c7ca1c3aa28e6243f0cd0040166ee9d369e5abe2ac1c610ed10be8b\": container with ID starting with ed35dc0a7c7ca1c3aa28e6243f0cd0040166ee9d369e5abe2ac1c610ed10be8b not found: ID does not exist" Oct 29 22:59:54.992168 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:54.992134 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t"] Oct 29 22:59:55.001462 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:55.001414 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t"] Oct 29 22:59:56.172922 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:56.172889 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a6b1fc-4fb1-41ae-99e2-359d9274c11d" path="/var/lib/kubelet/pods/d5a6b1fc-4fb1-41ae-99e2-359d9274c11d/volumes" Oct 29 22:59:57.430623 ip-10-0-130-252 kubenswrapper[4821]: I1029 22:59:57.430583 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27d16a-5c774b769f-76z4t" podUID="d5a6b1fc-4fb1-41ae-99e2-359d9274c11d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.99:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:00.172884 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.172852 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9"] Oct 29 23:00:00.173465 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.173450 4821 cpu_manager.go:414] "containerMap: removing container" podUID="d5a6b1fc-4fb1-41ae-99e2-359d9274c11d" containerName="maas-api" Oct 29 23:00:00.173513 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.173467 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a6b1fc-4fb1-41ae-99e2-359d9274c11d" containerName="maas-api" Oct 29 23:00:00.173607 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.173597 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a6b1fc-4fb1-41ae-99e2-359d9274c11d" containerName="maas-api" Oct 29 23:00:00.179406 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.179384 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4"] Oct 29 23:00:00.179548 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.179497 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9" Oct 29 23:00:00.183429 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.183409 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph"] Oct 29 23:00:00.183569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.183554 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4" Oct 29 23:00:00.193423 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.190031 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph" Oct 29 23:00:00.193423 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.193257 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb"] Oct 29 23:00:00.199463 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.199137 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb" Oct 29 23:00:00.201061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.201032 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4"] Oct 29 23:00:00.208346 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.208322 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9"] Oct 29 23:00:00.219417 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.217266 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6"] Oct 29 23:00:00.228888 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.227518 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6" Oct 29 23:00:00.229685 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.229661 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph"] Oct 29 23:00:00.236240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.235980 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8"] Oct 29 23:00:00.241382 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.241249 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8" Oct 29 23:00:00.245030 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.245006 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq"] Oct 29 23:00:00.250104 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.250081 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq" Oct 29 23:00:00.252642 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.252620 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb"] Oct 29 23:00:00.262142 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.259182 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6"] Oct 29 23:00:00.275193 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.275168 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq"] Oct 29 23:00:00.280836 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.280810 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8"] Oct 29 23:00:00.292489 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.292468 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:00:00.323417 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.323391 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82e062dd-2e5b-436b-b494-76382054e140-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4\" (UID: \"82e062dd-2e5b-436b-b494-76382054e140\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4" Oct 29 23:00:00.323573 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.323551 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm9vg\" (UniqueName: \"kubernetes.io/projected/13a1ca2f-190b-42d5-8ead-3d64c06111d2-kube-api-access-nm9vg\") pod \"maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9\" (UID: \"13a1ca2f-190b-42d5-8ead-3d64c06111d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9" Oct 29 23:00:00.323629 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.323581 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a1ca2f-190b-42d5-8ead-3d64c06111d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9\" (UID: \"13a1ca2f-190b-42d5-8ead-3d64c06111d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9" Oct 29 23:00:00.323685 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.323643 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5w2d\" (UniqueName: \"kubernetes.io/projected/a4b1e79c-0035-4bae-bc76-94d40a09c66c-kube-api-access-v5w2d\") pod \"maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph\" (UID: \"a4b1e79c-0035-4bae-bc76-94d40a09c66c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph" Oct 29 23:00:00.323729 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.323689 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcn8z\" (UniqueName: \"kubernetes.io/projected/15027c98-a748-4aab-b0f0-5d913bfd61ff-kube-api-access-wcn8z\") pod \"maas-api-key-cleanup-29888580-mrxrb\" (UID: \"15027c98-a748-4aab-b0f0-5d913bfd61ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb" Oct 29 23:00:00.323822 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.323805 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b1e79c-0035-4bae-bc76-94d40a09c66c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph\" (UID: \"a4b1e79c-0035-4bae-bc76-94d40a09c66c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph" Oct 29 23:00:00.323955 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.323941 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc62z\" (UniqueName: \"kubernetes.io/projected/82e062dd-2e5b-436b-b494-76382054e140-kube-api-access-wc62z\") pod \"maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4\" (UID: \"82e062dd-2e5b-436b-b494-76382054e140\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4" Oct 29 23:00:00.324024 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.323972 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15027c98-a748-4aab-b0f0-5d913bfd61ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888580-mrxrb\" (UID: \"15027c98-a748-4aab-b0f0-5d913bfd61ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb" Oct 29 23:00:00.425474 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.425386 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4br9\" (UniqueName: \"kubernetes.io/projected/63b44f7f-36f3-4f59-a45c-cd9af978b207-kube-api-access-h4br9\") pod \"maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6\" (UID: \"63b44f7f-36f3-4f59-a45c-cd9af978b207\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6" Oct 29 23:00:00.425474 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.425444 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82e062dd-2e5b-436b-b494-76382054e140-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4\" (UID: \"82e062dd-2e5b-436b-b494-76382054e140\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4" Oct 29 23:00:00.425666 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.425513 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7557d94-c264-4d93-847e-3e0c823acf34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq\" (UID: \"f7557d94-c264-4d93-847e-3e0c823acf34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq" Oct 29 23:00:00.425707 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.425695 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm9vg\" (UniqueName: \"kubernetes.io/projected/13a1ca2f-190b-42d5-8ead-3d64c06111d2-kube-api-access-nm9vg\") pod \"maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9\" (UID: \"13a1ca2f-190b-42d5-8ead-3d64c06111d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9" Oct 29 23:00:00.425739 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.425717 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a1ca2f-190b-42d5-8ead-3d64c06111d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9\" (UID: \"13a1ca2f-190b-42d5-8ead-3d64c06111d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9" Oct 29 23:00:00.425776 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.425759 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b44f7f-36f3-4f59-a45c-cd9af978b207-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6\" (UID: \"63b44f7f-36f3-4f59-a45c-cd9af978b207\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6" Oct 29 23:00:00.425837 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.425802 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfgcw\" (UniqueName: \"kubernetes.io/projected/f7557d94-c264-4d93-847e-3e0c823acf34-kube-api-access-zfgcw\") pod \"maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq\" (UID: \"f7557d94-c264-4d93-847e-3e0c823acf34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq" Oct 29 23:00:00.425892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.425831 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5w2d\" (UniqueName: \"kubernetes.io/projected/a4b1e79c-0035-4bae-bc76-94d40a09c66c-kube-api-access-v5w2d\") pod \"maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph\" (UID: \"a4b1e79c-0035-4bae-bc76-94d40a09c66c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph" Oct 29 23:00:00.425948 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.425875 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcn8z\" (UniqueName: \"kubernetes.io/projected/15027c98-a748-4aab-b0f0-5d913bfd61ff-kube-api-access-wcn8z\") pod \"maas-api-key-cleanup-29888580-mrxrb\" (UID: \"15027c98-a748-4aab-b0f0-5d913bfd61ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb" Oct 29 23:00:00.425948 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.425931 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b1e79c-0035-4bae-bc76-94d40a09c66c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph\" (UID: \"a4b1e79c-0035-4bae-bc76-94d40a09c66c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph" Oct 29 23:00:00.426064 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.425975 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v4cs\" (UniqueName: \"kubernetes.io/projected/c80b13fa-d61d-448a-9862-cd99080d88ea-kube-api-access-6v4cs\") pod \"maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8\" (UID: \"c80b13fa-d61d-448a-9862-cd99080d88ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8" Oct 29 23:00:00.426064 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.426048 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c80b13fa-d61d-448a-9862-cd99080d88ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8\" (UID: \"c80b13fa-d61d-448a-9862-cd99080d88ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8" Oct 29 23:00:00.426170 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.426081 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc62z\" (UniqueName: \"kubernetes.io/projected/82e062dd-2e5b-436b-b494-76382054e140-kube-api-access-wc62z\") pod \"maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4\" (UID: \"82e062dd-2e5b-436b-b494-76382054e140\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4" Oct 29 23:00:00.426170 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.426105 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15027c98-a748-4aab-b0f0-5d913bfd61ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888580-mrxrb\" (UID: \"15027c98-a748-4aab-b0f0-5d913bfd61ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb" Oct 29 23:00:00.426313 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.426244 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82e062dd-2e5b-436b-b494-76382054e140-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4\" (UID: \"82e062dd-2e5b-436b-b494-76382054e140\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4" Oct 29 23:00:00.426558 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.426535 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a1ca2f-190b-42d5-8ead-3d64c06111d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9\" (UID: \"13a1ca2f-190b-42d5-8ead-3d64c06111d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9" Oct 29 23:00:00.426718 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.426700 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b1e79c-0035-4bae-bc76-94d40a09c66c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph\" (UID: \"a4b1e79c-0035-4bae-bc76-94d40a09c66c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph" Oct 29 23:00:00.426894 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.426873 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15027c98-a748-4aab-b0f0-5d913bfd61ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888580-mrxrb\" (UID: \"15027c98-a748-4aab-b0f0-5d913bfd61ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb" Oct 29 23:00:00.449065 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.449034 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcn8z\" (UniqueName: \"kubernetes.io/projected/15027c98-a748-4aab-b0f0-5d913bfd61ff-kube-api-access-wcn8z\") pod \"maas-api-key-cleanup-29888580-mrxrb\" (UID: \"15027c98-a748-4aab-b0f0-5d913bfd61ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb" Oct 29 23:00:00.449239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.449038 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc62z\" (UniqueName: \"kubernetes.io/projected/82e062dd-2e5b-436b-b494-76382054e140-kube-api-access-wc62z\") pod \"maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4\" (UID: \"82e062dd-2e5b-436b-b494-76382054e140\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4" Oct 29 23:00:00.449239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.449091 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm9vg\" (UniqueName: \"kubernetes.io/projected/13a1ca2f-190b-42d5-8ead-3d64c06111d2-kube-api-access-nm9vg\") pod \"maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9\" (UID: \"13a1ca2f-190b-42d5-8ead-3d64c06111d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9" Oct 29 23:00:00.449359 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.449238 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5w2d\" (UniqueName: \"kubernetes.io/projected/a4b1e79c-0035-4bae-bc76-94d40a09c66c-kube-api-access-v5w2d\") pod \"maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph\" (UID: \"a4b1e79c-0035-4bae-bc76-94d40a09c66c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph" Oct 29 23:00:00.502916 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.502881 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9" Oct 29 23:00:00.527681 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.527643 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4br9\" (UniqueName: \"kubernetes.io/projected/63b44f7f-36f3-4f59-a45c-cd9af978b207-kube-api-access-h4br9\") pod \"maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6\" (UID: \"63b44f7f-36f3-4f59-a45c-cd9af978b207\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6" Oct 29 23:00:00.527810 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.527710 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7557d94-c264-4d93-847e-3e0c823acf34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq\" (UID: \"f7557d94-c264-4d93-847e-3e0c823acf34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq" Oct 29 23:00:00.527889 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.527870 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b44f7f-36f3-4f59-a45c-cd9af978b207-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6\" (UID: \"63b44f7f-36f3-4f59-a45c-cd9af978b207\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6" Oct 29 23:00:00.527958 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.527942 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfgcw\" (UniqueName: \"kubernetes.io/projected/f7557d94-c264-4d93-847e-3e0c823acf34-kube-api-access-zfgcw\") pod \"maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq\" (UID: \"f7557d94-c264-4d93-847e-3e0c823acf34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq" Oct 29 23:00:00.528020 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.528004 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v4cs\" (UniqueName: \"kubernetes.io/projected/c80b13fa-d61d-448a-9862-cd99080d88ea-kube-api-access-6v4cs\") pod \"maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8\" (UID: \"c80b13fa-d61d-448a-9862-cd99080d88ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8" Oct 29 23:00:00.528081 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.528066 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c80b13fa-d61d-448a-9862-cd99080d88ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8\" (UID: \"c80b13fa-d61d-448a-9862-cd99080d88ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8" Oct 29 23:00:00.528458 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.528433 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7557d94-c264-4d93-847e-3e0c823acf34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq\" (UID: \"f7557d94-c264-4d93-847e-3e0c823acf34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq" Oct 29 23:00:00.528655 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.528635 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b44f7f-36f3-4f59-a45c-cd9af978b207-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6\" (UID: \"63b44f7f-36f3-4f59-a45c-cd9af978b207\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6" Oct 29 23:00:00.528738 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.528715 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c80b13fa-d61d-448a-9862-cd99080d88ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8\" (UID: \"c80b13fa-d61d-448a-9862-cd99080d88ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8" Oct 29 23:00:00.533888 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.533868 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4" Oct 29 23:00:00.545188 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.545168 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb" Oct 29 23:00:00.555410 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.551952 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfgcw\" (UniqueName: \"kubernetes.io/projected/f7557d94-c264-4d93-847e-3e0c823acf34-kube-api-access-zfgcw\") pod \"maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq\" (UID: \"f7557d94-c264-4d93-847e-3e0c823acf34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq" Oct 29 23:00:00.555410 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.553049 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4br9\" (UniqueName: \"kubernetes.io/projected/63b44f7f-36f3-4f59-a45c-cd9af978b207-kube-api-access-h4br9\") pod \"maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6\" (UID: \"63b44f7f-36f3-4f59-a45c-cd9af978b207\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6" Oct 29 23:00:00.556823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.556798 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v4cs\" (UniqueName: \"kubernetes.io/projected/c80b13fa-d61d-448a-9862-cd99080d88ea-kube-api-access-6v4cs\") pod \"maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8\" (UID: \"c80b13fa-d61d-448a-9862-cd99080d88ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8" Oct 29 23:00:00.574485 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.574168 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph" Oct 29 23:00:00.583154 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.582988 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6" Oct 29 23:00:00.598097 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.598064 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8" Oct 29 23:00:00.600228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.600150 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq" Oct 29 23:00:00.832563 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.832535 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9"] Oct 29 23:00:00.834577 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:00:00.834546 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a1ca2f_190b_42d5_8ead_3d64c06111d2.slice/crio-66cdc22bfa1f2189008738029ac7d1386f31cabb07c5a7cd71b8aba4b7afd4ae WatchSource:0}: Error finding container 66cdc22bfa1f2189008738029ac7d1386f31cabb07c5a7cd71b8aba4b7afd4ae: Status 404 returned error can't find the container with id 66cdc22bfa1f2189008738029ac7d1386f31cabb07c5a7cd71b8aba4b7afd4ae Oct 29 23:00:00.902213 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.902166 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb"] Oct 29 23:00:00.904072 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:00:00.903484 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15027c98_a748_4aab_b0f0_5d913bfd61ff.slice/crio-8a2d554e564b8d413201ce5a0ce4293bb0dc4349af2898f74b2794bb1006c0a9 WatchSource:0}: Error finding container 8a2d554e564b8d413201ce5a0ce4293bb0dc4349af2898f74b2794bb1006c0a9: Status 404 returned error can't find the container with id 8a2d554e564b8d413201ce5a0ce4293bb0dc4349af2898f74b2794bb1006c0a9 Oct 29 23:00:00.904376 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.904352 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs" Oct 29 23:00:00.907498 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:00.907472 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t" Oct 29 23:00:01.028055 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:01.028003 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9" event={"ID":"13a1ca2f-190b-42d5-8ead-3d64c06111d2","Type":"ContainerStarted","Data":"66cdc22bfa1f2189008738029ac7d1386f31cabb07c5a7cd71b8aba4b7afd4ae"} Oct 29 23:00:01.030789 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:01.030755 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb" event={"ID":"15027c98-a748-4aab-b0f0-5d913bfd61ff","Type":"ContainerStarted","Data":"8a2d554e564b8d413201ce5a0ce4293bb0dc4349af2898f74b2794bb1006c0a9"} Oct 29 23:00:01.078860 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:01.078831 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4"] Oct 29 23:00:01.119020 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:00:01.118985 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e062dd_2e5b_436b_b494_76382054e140.slice/crio-a1857047143719d44fc97920a51d626225e072e8967b3a4c8c6404c3b844ea7e WatchSource:0}: Error finding container a1857047143719d44fc97920a51d626225e072e8967b3a4c8c6404c3b844ea7e: Status 404 returned error can't find the container with id a1857047143719d44fc97920a51d626225e072e8967b3a4c8c6404c3b844ea7e Oct 29 23:00:01.128872 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:01.128848 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph"] Oct 29 23:00:01.132412 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:00:01.132389 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b1e79c_0035_4bae_bc76_94d40a09c66c.slice/crio-f09321ad8f5e68ac1157342e1b7cfc391bfc00b51a7ac889893f2469d3dda8f9 WatchSource:0}: Error finding container f09321ad8f5e68ac1157342e1b7cfc391bfc00b51a7ac889893f2469d3dda8f9: Status 404 returned error can't find the container with id f09321ad8f5e68ac1157342e1b7cfc391bfc00b51a7ac889893f2469d3dda8f9 Oct 29 23:00:01.204928 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:01.204896 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8"] Oct 29 23:00:01.205412 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:00:01.205385 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80b13fa_d61d_448a_9862_cd99080d88ea.slice/crio-19387cac20947a83deeafec8de30edb45a2f10ee41069c402ebee6a2084ced58 WatchSource:0}: Error finding container 19387cac20947a83deeafec8de30edb45a2f10ee41069c402ebee6a2084ced58: Status 404 returned error can't find the container with id 19387cac20947a83deeafec8de30edb45a2f10ee41069c402ebee6a2084ced58 Oct 29 23:00:01.222689 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:01.221301 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6"] Oct 29 23:00:01.227944 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:01.227921 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq"] Oct 29 23:00:01.319839 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:00:01.319800 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b44f7f_36f3_4f59_a45c_cd9af978b207.slice/crio-7ee84f7bb957d4ad0a3712a4b23bac7048a13de981a4856e568e0c731838a886 WatchSource:0}: Error finding container 7ee84f7bb957d4ad0a3712a4b23bac7048a13de981a4856e568e0c731838a886: Status 404 returned error can't find the container with id 7ee84f7bb957d4ad0a3712a4b23bac7048a13de981a4856e568e0c731838a886 Oct 29 23:00:01.322941 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:00:01.322915 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7557d94_c264_4d93_847e_3e0c823acf34.slice/crio-b2ca1009266a5ded4d1bc67a84f7b70ef0df5f1e7b6b7ffb63a972c51eeef28f WatchSource:0}: Error finding container b2ca1009266a5ded4d1bc67a84f7b70ef0df5f1e7b6b7ffb63a972c51eeef28f: Status 404 returned error can't find the container with id b2ca1009266a5ded4d1bc67a84f7b70ef0df5f1e7b6b7ffb63a972c51eeef28f Oct 29 23:00:02.041691 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:02.041663 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8" event={"ID":"c80b13fa-d61d-448a-9862-cd99080d88ea","Type":"ContainerStarted","Data":"14771963bfa27aaa4cc258bbe51a71259dffa5cd7b883d9b50935dd54fac0f5c"} Oct 29 23:00:02.041804 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:02.041718 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8" event={"ID":"c80b13fa-d61d-448a-9862-cd99080d88ea","Type":"ContainerStarted","Data":"19387cac20947a83deeafec8de30edb45a2f10ee41069c402ebee6a2084ced58"} Oct 29 23:00:02.043334 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:02.043308 4821 generic.go:356] "Generic (PLEG): container finished" podID="13a1ca2f-190b-42d5-8ead-3d64c06111d2" containerID="0b85fd7d5a5fd21244e8c9478d1d34346d8ab6c1102db69dc897eb3cae651111" exitCode=60 Oct 29 23:00:02.043434 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:02.043386 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9" event={"ID":"13a1ca2f-190b-42d5-8ead-3d64c06111d2","Type":"ContainerDied","Data":"0b85fd7d5a5fd21244e8c9478d1d34346d8ab6c1102db69dc897eb3cae651111"} Oct 29 23:00:02.043661 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:02.043646 4821 scope.go:122] "RemoveContainer" containerID="0b85fd7d5a5fd21244e8c9478d1d34346d8ab6c1102db69dc897eb3cae651111" Oct 29 23:00:02.045171 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:02.045148 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6" event={"ID":"63b44f7f-36f3-4f59-a45c-cd9af978b207","Type":"ContainerStarted","Data":"a2566ad982185b65d1f9f993f5e7d056eda2aaca05a4cd762983b9b5d2a7e9fd"} Oct 29 23:00:02.045297 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:02.045237 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6" event={"ID":"63b44f7f-36f3-4f59-a45c-cd9af978b207","Type":"ContainerStarted","Data":"7ee84f7bb957d4ad0a3712a4b23bac7048a13de981a4856e568e0c731838a886"} Oct 29 23:00:02.046948 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:02.046922 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph" event={"ID":"a4b1e79c-0035-4bae-bc76-94d40a09c66c","Type":"ContainerStarted","Data":"3b88628df4a10ff4d3af6d18ccebe1baf4ef9edb462fbde76f52b3229d3dda69"} Oct 29 23:00:02.047047 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:02.046968 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph" event={"ID":"a4b1e79c-0035-4bae-bc76-94d40a09c66c","Type":"ContainerStarted","Data":"f09321ad8f5e68ac1157342e1b7cfc391bfc00b51a7ac889893f2469d3dda8f9"} Oct 29 23:00:02.048663 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:02.048642 4821 generic.go:356] "Generic (PLEG): container finished" podID="15027c98-a748-4aab-b0f0-5d913bfd61ff" containerID="b9a318437e2ff09ffc5c45c199a717327c0d0e840379a36bbf5bfbc86ffd2b4a" exitCode=60 Oct 29 23:00:02.048744 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:02.048720 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb" event={"ID":"15027c98-a748-4aab-b0f0-5d913bfd61ff","Type":"ContainerDied","Data":"b9a318437e2ff09ffc5c45c199a717327c0d0e840379a36bbf5bfbc86ffd2b4a"} Oct 29 23:00:02.048953 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:02.048939 4821 scope.go:122] "RemoveContainer" containerID="b9a318437e2ff09ffc5c45c199a717327c0d0e840379a36bbf5bfbc86ffd2b4a" Oct 29 23:00:02.050329 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:02.050307 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq" event={"ID":"f7557d94-c264-4d93-847e-3e0c823acf34","Type":"ContainerStarted","Data":"c2f73a5b0cfe58542c51125a9836c5f3552c227d4b24337443a946f7b7815fcd"} Oct 29 23:00:02.050421 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:02.050354 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq" event={"ID":"f7557d94-c264-4d93-847e-3e0c823acf34","Type":"ContainerStarted","Data":"b2ca1009266a5ded4d1bc67a84f7b70ef0df5f1e7b6b7ffb63a972c51eeef28f"} Oct 29 23:00:02.051909 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:02.051889 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4" event={"ID":"82e062dd-2e5b-436b-b494-76382054e140","Type":"ContainerStarted","Data":"908d37e2ac37b9950f89a53d72856272812205bf14c6049094449410db57a3a1"} Oct 29 23:00:02.051997 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:02.051924 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4" event={"ID":"82e062dd-2e5b-436b-b494-76382054e140","Type":"ContainerStarted","Data":"a1857047143719d44fc97920a51d626225e072e8967b3a4c8c6404c3b844ea7e"} Oct 29 23:00:02.080671 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:02.080625 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8" podStartSLOduration=2.080610099 podStartE2EDuration="2.080610099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.080302534 +0000 UTC m=+1056.020850443" watchObservedRunningTime="2026-10-29 23:00:02.080610099 +0000 UTC m=+1056.021158009" Oct 29 23:00:02.122223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:02.122145 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6" podStartSLOduration=2.122130899 podStartE2EDuration="2.122130899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.118796271 +0000 UTC m=+1056.059344182" watchObservedRunningTime="2026-10-29 23:00:02.122130899 +0000 UTC m=+1056.062678809" Oct 29 23:00:02.161675 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:02.161615 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq" podStartSLOduration=2.161599885 podStartE2EDuration="2.161599885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.157474884 +0000 UTC m=+1056.098022795" watchObservedRunningTime="2026-10-29 23:00:02.161599885 +0000 UTC m=+1056.102147794" Oct 29 23:00:02.199191 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:02.199094 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4" podStartSLOduration=2.199079714 podStartE2EDuration="2.199079714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.195951797 +0000 UTC m=+1056.136499706" watchObservedRunningTime="2026-10-29 23:00:02.199079714 +0000 UTC m=+1056.139627642" Oct 29 23:00:02.237774 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:02.237717 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph" podStartSLOduration=2.237702126 podStartE2EDuration="2.237702126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.234663234 +0000 UTC m=+1056.175211144" watchObservedRunningTime="2026-10-29 23:00:02.237702126 +0000 UTC m=+1056.178250035" Oct 29 23:00:03.061090 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.060999 4821 generic.go:356] "Generic (PLEG): container finished" podID="a4b1e79c-0035-4bae-bc76-94d40a09c66c" containerID="3b88628df4a10ff4d3af6d18ccebe1baf4ef9edb462fbde76f52b3229d3dda69" exitCode=60 Oct 29 23:00:03.061285 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.061095 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph" event={"ID":"a4b1e79c-0035-4bae-bc76-94d40a09c66c","Type":"ContainerDied","Data":"3b88628df4a10ff4d3af6d18ccebe1baf4ef9edb462fbde76f52b3229d3dda69"} Oct 29 23:00:03.061503 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.061485 4821 scope.go:122] "RemoveContainer" containerID="3b88628df4a10ff4d3af6d18ccebe1baf4ef9edb462fbde76f52b3229d3dda69" Oct 29 23:00:03.063427 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.063363 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb" event={"ID":"15027c98-a748-4aab-b0f0-5d913bfd61ff","Type":"ContainerStarted","Data":"1fc0fd4d2c5f75a5f750e4a678c02420deae6aea6c6a74c34236787cf8f62b05"} Oct 29 23:00:03.065042 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.065021 4821 generic.go:356] "Generic (PLEG): container finished" podID="f7557d94-c264-4d93-847e-3e0c823acf34" containerID="c2f73a5b0cfe58542c51125a9836c5f3552c227d4b24337443a946f7b7815fcd" exitCode=60 Oct 29 23:00:03.065136 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.065096 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq" event={"ID":"f7557d94-c264-4d93-847e-3e0c823acf34","Type":"ContainerDied","Data":"c2f73a5b0cfe58542c51125a9836c5f3552c227d4b24337443a946f7b7815fcd"} Oct 29 23:00:03.065418 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.065401 4821 scope.go:122] "RemoveContainer" containerID="c2f73a5b0cfe58542c51125a9836c5f3552c227d4b24337443a946f7b7815fcd" Oct 29 23:00:03.066635 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.066611 4821 generic.go:356] "Generic (PLEG): container finished" podID="82e062dd-2e5b-436b-b494-76382054e140" containerID="908d37e2ac37b9950f89a53d72856272812205bf14c6049094449410db57a3a1" exitCode=60 Oct 29 23:00:03.066722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.066706 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4" event={"ID":"82e062dd-2e5b-436b-b494-76382054e140","Type":"ContainerDied","Data":"908d37e2ac37b9950f89a53d72856272812205bf14c6049094449410db57a3a1"} Oct 29 23:00:03.066911 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.066884 4821 scope.go:122] "RemoveContainer" containerID="908d37e2ac37b9950f89a53d72856272812205bf14c6049094449410db57a3a1" Oct 29 23:00:03.070763 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.070738 4821 generic.go:356] "Generic (PLEG): container finished" podID="c80b13fa-d61d-448a-9862-cd99080d88ea" containerID="14771963bfa27aaa4cc258bbe51a71259dffa5cd7b883d9b50935dd54fac0f5c" exitCode=60 Oct 29 23:00:03.070851 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.070833 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8" event={"ID":"c80b13fa-d61d-448a-9862-cd99080d88ea","Type":"ContainerDied","Data":"14771963bfa27aaa4cc258bbe51a71259dffa5cd7b883d9b50935dd54fac0f5c"} Oct 29 23:00:03.071084 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.071064 4821 scope.go:122] "RemoveContainer" containerID="14771963bfa27aaa4cc258bbe51a71259dffa5cd7b883d9b50935dd54fac0f5c" Oct 29 23:00:03.072996 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.072975 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9" event={"ID":"13a1ca2f-190b-42d5-8ead-3d64c06111d2","Type":"ContainerStarted","Data":"3d5a53acc7f3563e87d2105cb3739f7fe710e385a4148821449389bce739a517"} Oct 29 23:00:03.077609 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.075386 4821 generic.go:356] "Generic (PLEG): container finished" podID="63b44f7f-36f3-4f59-a45c-cd9af978b207" containerID="a2566ad982185b65d1f9f993f5e7d056eda2aaca05a4cd762983b9b5d2a7e9fd" exitCode=60 Oct 29 23:00:03.077609 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.075428 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6" event={"ID":"63b44f7f-36f3-4f59-a45c-cd9af978b207","Type":"ContainerDied","Data":"a2566ad982185b65d1f9f993f5e7d056eda2aaca05a4cd762983b9b5d2a7e9fd"} Oct 29 23:00:03.077609 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.075713 4821 scope.go:122] "RemoveContainer" containerID="a2566ad982185b65d1f9f993f5e7d056eda2aaca05a4cd762983b9b5d2a7e9fd" Oct 29 23:00:03.264357 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.264296 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb" podStartSLOduration=3.264279684 podStartE2EDuration="3.264279684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.261748945 +0000 UTC m=+1057.202296855" watchObservedRunningTime="2026-10-29 23:00:03.264279684 +0000 UTC m=+1057.204827607" Oct 29 23:00:03.306126 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.306074 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9" podStartSLOduration=3.30605889 podStartE2EDuration="3.30605889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.300826185 +0000 UTC m=+1057.241374086" watchObservedRunningTime="2026-10-29 23:00:03.30605889 +0000 UTC m=+1057.246606799" Oct 29 23:00:03.480036 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.479984 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45"] Oct 29 23:00:03.480345 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.480283 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" podUID="525944e9-324d-4398-88bb-eeeea15478ce" containerName="istio-proxy" containerID="cri-o://6ee2a527c5334137142f8768f3796a398a0b7798d68eb0d02b21faa44f1b9341" gracePeriod=30 Oct 29 23:00:03.760886 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.760851 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk"] Oct 29 23:00:03.761069 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.761047 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk" podUID="bdbff57a-f530-441a-b708-8e3d2c64ed13" containerName="maas-api" containerID="cri-o://19da036f3264d37334120b91f5302185fc7bf04a57cd068d67843504062acdb7" gracePeriod=30 Oct 29 23:00:03.906313 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.884285 4821 patch_prober.go:29] interesting pod/e2e-api-b-27d16a-openshift-default-869755f997-p2h45 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:00:03.906313 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:03.884363 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" podUID="525944e9-324d-4398-88bb-eeeea15478ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:00:04.132597 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.132547 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph" event={"ID":"a4b1e79c-0035-4bae-bc76-94d40a09c66c","Type":"ContainerStarted","Data":"989050deb63aebb9a62b44233dba4253e5f573e46e63fd1cc6fe34eb0ee4f297"} Oct 29 23:00:04.143220 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.141804 4821 generic.go:356] "Generic (PLEG): container finished" podID="15027c98-a748-4aab-b0f0-5d913bfd61ff" containerID="1fc0fd4d2c5f75a5f750e4a678c02420deae6aea6c6a74c34236787cf8f62b05" exitCode=60 Oct 29 23:00:04.143220 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.141935 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb" event={"ID":"15027c98-a748-4aab-b0f0-5d913bfd61ff","Type":"ContainerDied","Data":"1fc0fd4d2c5f75a5f750e4a678c02420deae6aea6c6a74c34236787cf8f62b05"} Oct 29 23:00:04.143220 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.141994 4821 scope.go:122] "RemoveContainer" containerID="b9a318437e2ff09ffc5c45c199a717327c0d0e840379a36bbf5bfbc86ffd2b4a" Oct 29 23:00:04.143220 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.142522 4821 scope.go:122] "RemoveContainer" containerID="1fc0fd4d2c5f75a5f750e4a678c02420deae6aea6c6a74c34236787cf8f62b05" Oct 29 23:00:04.143220 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:04.142955 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888580-mrxrb_odh-ai-gateway-infra(15027c98-a748-4aab-b0f0-5d913bfd61ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb" podUID="15027c98-a748-4aab-b0f0-5d913bfd61ff" Oct 29 23:00:04.150415 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.150387 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq" event={"ID":"f7557d94-c264-4d93-847e-3e0c823acf34","Type":"ContainerStarted","Data":"9327b2daa78c7949cc356e905c7498129f17c6a680122268dc3f4f85644d2d65"} Oct 29 23:00:04.155621 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.155586 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4" event={"ID":"82e062dd-2e5b-436b-b494-76382054e140","Type":"ContainerStarted","Data":"7350dfa0fc6acf675c2f6b3538d8088fb3cbc9acbe91237a0f85d24971656549"} Oct 29 23:00:04.172518 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.172350 4821 generic.go:356] "Generic (PLEG): container finished" podID="bdbff57a-f530-441a-b708-8e3d2c64ed13" containerID="19da036f3264d37334120b91f5302185fc7bf04a57cd068d67843504062acdb7" exitCode=0 Oct 29 23:00:04.266443 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.261969 4821 generic.go:356] "Generic (PLEG): container finished" podID="13a1ca2f-190b-42d5-8ead-3d64c06111d2" containerID="3d5a53acc7f3563e87d2105cb3739f7fe710e385a4148821449389bce739a517" exitCode=60 Oct 29 23:00:04.266443 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.265339 4821 scope.go:122] "RemoveContainer" containerID="3d5a53acc7f3563e87d2105cb3739f7fe710e385a4148821449389bce739a517" Oct 29 23:00:04.266443 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:04.266190 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9_odh-ai-gateway-infra(13a1ca2f-190b-42d5-8ead-3d64c06111d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9" podUID="13a1ca2f-190b-42d5-8ead-3d64c06111d2" Oct 29 23:00:04.391380 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.387246 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk" event={"ID":"bdbff57a-f530-441a-b708-8e3d2c64ed13","Type":"ContainerDied","Data":"19da036f3264d37334120b91f5302185fc7bf04a57cd068d67843504062acdb7"} Oct 29 23:00:04.391380 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.387322 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8" event={"ID":"c80b13fa-d61d-448a-9862-cd99080d88ea","Type":"ContainerStarted","Data":"478afa57a478b85cacf9f3a4f991fe0df4a7adf9d7c4b3041d5776a13a2e3fba"} Oct 29 23:00:04.391380 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.387351 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9" event={"ID":"13a1ca2f-190b-42d5-8ead-3d64c06111d2","Type":"ContainerDied","Data":"3d5a53acc7f3563e87d2105cb3739f7fe710e385a4148821449389bce739a517"} Oct 29 23:00:04.391380 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.387401 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6" event={"ID":"63b44f7f-36f3-4f59-a45c-cd9af978b207","Type":"ContainerStarted","Data":"c80c4cb214e6b7824b39d27cc5b4afa5e0d58fb401938b0f01c9b1ea8ae11702"} Oct 29 23:00:04.391380 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.387476 4821 scope.go:122] "RemoveContainer" containerID="0b85fd7d5a5fd21244e8c9478d1d34346d8ab6c1102db69dc897eb3cae651111" Oct 29 23:00:04.500362 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.500317 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7608-f16f60"] Oct 29 23:00:04.535076 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.534182 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-f16f60" Oct 29 23:00:04.540890 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.536313 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7608-f16f60"] Oct 29 23:00:04.549243 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.549191 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fvr2l\"" Oct 29 23:00:04.556139 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.554740 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vm4q\" (UniqueName: \"kubernetes.io/projected/3dbba931-b65f-4e62-aed3-35759a78d6d2-kube-api-access-4vm4q\") pod \"test-curl-7608-f16f60\" (UID: \"3dbba931-b65f-4e62-aed3-35759a78d6d2\") " pod="opendatahub/test-curl-7608-f16f60" Oct 29 23:00:04.578393 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.578368 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk" Oct 29 23:00:04.656820 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.656789 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdbff57a-f530-441a-b708-8e3d2c64ed13-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdbff57a-f530-441a-b708-8e3d2c64ed13-maas-api-tls\") pod \"bdbff57a-f530-441a-b708-8e3d2c64ed13\" (UID: \"bdbff57a-f530-441a-b708-8e3d2c64ed13\") " Oct 29 23:00:04.659364 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.659329 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdbff57a-f530-441a-b708-8e3d2c64ed13-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdbff57a-f530-441a-b708-8e3d2c64ed13-openshift-service-ca\") pod \"bdbff57a-f530-441a-b708-8e3d2c64ed13\" (UID: \"bdbff57a-f530-441a-b708-8e3d2c64ed13\") " Oct 29 23:00:04.659583 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.659463 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdbff57a-f530-441a-b708-8e3d2c64ed13-kube-api-access-5ls2n\" (UniqueName: \"kubernetes.io/projected/bdbff57a-f530-441a-b708-8e3d2c64ed13-kube-api-access-5ls2n\") pod \"bdbff57a-f530-441a-b708-8e3d2c64ed13\" (UID: \"bdbff57a-f530-441a-b708-8e3d2c64ed13\") " Oct 29 23:00:04.659583 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.659576 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdbff57a-f530-441a-b708-8e3d2c64ed13-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdbff57a-f530-441a-b708-8e3d2c64ed13-metrics-certs\") pod \"bdbff57a-f530-441a-b708-8e3d2c64ed13\" (UID: \"bdbff57a-f530-441a-b708-8e3d2c64ed13\") " Oct 29 23:00:04.659992 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.659967 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vm4q\" (UniqueName: \"kubernetes.io/projected/3dbba931-b65f-4e62-aed3-35759a78d6d2-kube-api-access-4vm4q\") pod \"test-curl-7608-f16f60\" (UID: \"3dbba931-b65f-4e62-aed3-35759a78d6d2\") " pod="opendatahub/test-curl-7608-f16f60" Oct 29 23:00:04.660547 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.660520 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdbff57a-f530-441a-b708-8e3d2c64ed13-maas-api-tls" pod "bdbff57a-f530-441a-b708-8e3d2c64ed13" (UID: "bdbff57a-f530-441a-b708-8e3d2c64ed13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:04.660883 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.660857 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdbff57a-f530-441a-b708-8e3d2c64ed13-openshift-service-ca" pod "bdbff57a-f530-441a-b708-8e3d2c64ed13" (UID: "bdbff57a-f530-441a-b708-8e3d2c64ed13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:04.664074 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.663959 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdbff57a-f530-441a-b708-8e3d2c64ed13-metrics-certs" pod "bdbff57a-f530-441a-b708-8e3d2c64ed13" (UID: "bdbff57a-f530-441a-b708-8e3d2c64ed13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:04.675484 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.664590 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbff57a-f530-441a-b708-8e3d2c64ed13-kube-api-access-5ls2n" pod "bdbff57a-f530-441a-b708-8e3d2c64ed13" (UID: "bdbff57a-f530-441a-b708-8e3d2c64ed13"). InnerVolumeSpecName "kube-api-access-5ls2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:04.685580 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.685547 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vm4q\" (UniqueName: \"kubernetes.io/projected/3dbba931-b65f-4e62-aed3-35759a78d6d2-kube-api-access-4vm4q\") pod \"test-curl-7608-f16f60\" (UID: \"3dbba931-b65f-4e62-aed3-35759a78d6d2\") " pod="opendatahub/test-curl-7608-f16f60" Oct 29 23:00:04.772659 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.772499 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ls2n\" (UniqueName: \"kubernetes.io/projected/bdbff57a-f530-441a-b708-8e3d2c64ed13-kube-api-access-5ls2n\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:04.772659 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.772535 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdbff57a-f530-441a-b708-8e3d2c64ed13-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:04.772659 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.772545 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdbff57a-f530-441a-b708-8e3d2c64ed13-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:04.772659 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.772553 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdbff57a-f530-441a-b708-8e3d2c64ed13-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:04.900238 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:04.899480 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-f16f60" Oct 29 23:00:05.255839 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.255787 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7608-f16f60"] Oct 29 23:00:05.258703 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:00:05.258673 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3dbba931_b65f_4e62_aed3_35759a78d6d2.slice/crio-fa1e37ca00c99afe62a16ac70d315e9325ee4e24a6a711b29627f9cb85561519 WatchSource:0}: Error finding container fa1e37ca00c99afe62a16ac70d315e9325ee4e24a6a711b29627f9cb85561519: Status 404 returned error can't find the container with id fa1e37ca00c99afe62a16ac70d315e9325ee4e24a6a711b29627f9cb85561519 Oct 29 23:00:05.326489 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.326346 4821 scope.go:122] "RemoveContainer" containerID="3d5a53acc7f3563e87d2105cb3739f7fe710e385a4148821449389bce739a517" Oct 29 23:00:05.326879 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:05.326797 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9_odh-ai-gateway-infra(13a1ca2f-190b-42d5-8ead-3d64c06111d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9" podUID="13a1ca2f-190b-42d5-8ead-3d64c06111d2" Oct 29 23:00:05.330676 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.330640 4821 generic.go:356] "Generic (PLEG): container finished" podID="63b44f7f-36f3-4f59-a45c-cd9af978b207" containerID="c80c4cb214e6b7824b39d27cc5b4afa5e0d58fb401938b0f01c9b1ea8ae11702" exitCode=60 Oct 29 23:00:05.331282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.331085 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6" event={"ID":"63b44f7f-36f3-4f59-a45c-cd9af978b207","Type":"ContainerDied","Data":"c80c4cb214e6b7824b39d27cc5b4afa5e0d58fb401938b0f01c9b1ea8ae11702"} Oct 29 23:00:05.331282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.331152 4821 scope.go:122] "RemoveContainer" containerID="a2566ad982185b65d1f9f993f5e7d056eda2aaca05a4cd762983b9b5d2a7e9fd" Oct 29 23:00:05.331833 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.331816 4821 scope.go:122] "RemoveContainer" containerID="c80c4cb214e6b7824b39d27cc5b4afa5e0d58fb401938b0f01c9b1ea8ae11702" Oct 29 23:00:05.332425 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:05.332388 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6_odh-ai-gateway-infra(63b44f7f-36f3-4f59-a45c-cd9af978b207)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6" podUID="63b44f7f-36f3-4f59-a45c-cd9af978b207" Oct 29 23:00:05.350502 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.349873 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7608-f16f60" event={"ID":"3dbba931-b65f-4e62-aed3-35759a78d6d2","Type":"ContainerStarted","Data":"fa1e37ca00c99afe62a16ac70d315e9325ee4e24a6a711b29627f9cb85561519"} Oct 29 23:00:05.357073 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.356528 4821 generic.go:356] "Generic (PLEG): container finished" podID="a4b1e79c-0035-4bae-bc76-94d40a09c66c" containerID="989050deb63aebb9a62b44233dba4253e5f573e46e63fd1cc6fe34eb0ee4f297" exitCode=60 Oct 29 23:00:05.357073 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.356663 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph" event={"ID":"a4b1e79c-0035-4bae-bc76-94d40a09c66c","Type":"ContainerDied","Data":"989050deb63aebb9a62b44233dba4253e5f573e46e63fd1cc6fe34eb0ee4f297"} Oct 29 23:00:05.357314 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.357132 4821 scope.go:122] "RemoveContainer" containerID="989050deb63aebb9a62b44233dba4253e5f573e46e63fd1cc6fe34eb0ee4f297" Oct 29 23:00:05.357720 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:05.357619 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph_odh-ai-gateway-infra(a4b1e79c-0035-4bae-bc76-94d40a09c66c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph" podUID="a4b1e79c-0035-4bae-bc76-94d40a09c66c" Oct 29 23:00:05.364506 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.360156 4821 scope.go:122] "RemoveContainer" containerID="1fc0fd4d2c5f75a5f750e4a678c02420deae6aea6c6a74c34236787cf8f62b05" Oct 29 23:00:05.364506 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:05.360580 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888580-mrxrb_odh-ai-gateway-infra(15027c98-a748-4aab-b0f0-5d913bfd61ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb" podUID="15027c98-a748-4aab-b0f0-5d913bfd61ff" Oct 29 23:00:05.406422 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.406395 4821 scope.go:122] "RemoveContainer" containerID="3b88628df4a10ff4d3af6d18ccebe1baf4ef9edb462fbde76f52b3229d3dda69" Oct 29 23:00:05.470223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.462284 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd"] Oct 29 23:00:05.470223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.463388 4821 cpu_manager.go:414] "containerMap: removing container" podUID="bdbff57a-f530-441a-b708-8e3d2c64ed13" containerName="maas-api" Oct 29 23:00:05.470223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.463408 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdbff57a-f530-441a-b708-8e3d2c64ed13" containerName="maas-api" Oct 29 23:00:05.470223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.463591 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdbff57a-f530-441a-b708-8e3d2c64ed13" containerName="maas-api" Oct 29 23:00:05.472492 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.472461 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.474527 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.474488 4821 generic.go:356] "Generic (PLEG): container finished" podID="f7557d94-c264-4d93-847e-3e0c823acf34" containerID="9327b2daa78c7949cc356e905c7498129f17c6a680122268dc3f4f85644d2d65" exitCode=60 Oct 29 23:00:05.474795 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.474769 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq" event={"ID":"f7557d94-c264-4d93-847e-3e0c823acf34","Type":"ContainerDied","Data":"9327b2daa78c7949cc356e905c7498129f17c6a680122268dc3f4f85644d2d65"} Oct 29 23:00:05.475013 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.474975 4821 scope.go:122] "RemoveContainer" containerID="c2f73a5b0cfe58542c51125a9836c5f3552c227d4b24337443a946f7b7815fcd" Oct 29 23:00:05.475818 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.475803 4821 scope.go:122] "RemoveContainer" containerID="9327b2daa78c7949cc356e905c7498129f17c6a680122268dc3f4f85644d2d65" Oct 29 23:00:05.476576 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:05.476554 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq_odh-ai-gateway-infra(f7557d94-c264-4d93-847e-3e0c823acf34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq" podUID="f7557d94-c264-4d93-847e-3e0c823acf34" Oct 29 23:00:05.497718 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.497681 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd"] Oct 29 23:00:05.509130 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.500830 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f173b1-openshift-default-dockercfg-92xxg\"" Oct 29 23:00:05.523100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.523072 4821 generic.go:356] "Generic (PLEG): container finished" podID="82e062dd-2e5b-436b-b494-76382054e140" containerID="7350dfa0fc6acf675c2f6b3538d8088fb3cbc9acbe91237a0f85d24971656549" exitCode=60 Oct 29 23:00:05.523997 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.523823 4821 scope.go:122] "RemoveContainer" containerID="7350dfa0fc6acf675c2f6b3538d8088fb3cbc9acbe91237a0f85d24971656549" Oct 29 23:00:05.534674 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.523845 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4" event={"ID":"82e062dd-2e5b-436b-b494-76382054e140","Type":"ContainerDied","Data":"7350dfa0fc6acf675c2f6b3538d8088fb3cbc9acbe91237a0f85d24971656549"} Oct 29 23:00:05.534906 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.534886 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk" event={"ID":"bdbff57a-f530-441a-b708-8e3d2c64ed13","Type":"ContainerDied","Data":"4fcf538b6804cab67b98e6eeb085b76eb795ad7f2028a09f5edaeb2525bf870f"} Oct 29 23:00:05.535041 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.529788 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk" Oct 29 23:00:05.540397 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:05.540367 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4_odh-ai-gateway-infra(82e062dd-2e5b-436b-b494-76382054e140)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4" podUID="82e062dd-2e5b-436b-b494-76382054e140" Oct 29 23:00:05.542571 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.542545 4821 generic.go:356] "Generic (PLEG): container finished" podID="c80b13fa-d61d-448a-9862-cd99080d88ea" containerID="478afa57a478b85cacf9f3a4f991fe0df4a7adf9d7c4b3041d5776a13a2e3fba" exitCode=60 Oct 29 23:00:05.542740 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.542721 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8" event={"ID":"c80b13fa-d61d-448a-9862-cd99080d88ea","Type":"ContainerDied","Data":"478afa57a478b85cacf9f3a4f991fe0df4a7adf9d7c4b3041d5776a13a2e3fba"} Oct 29 23:00:05.543999 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.543370 4821 scope.go:122] "RemoveContainer" containerID="478afa57a478b85cacf9f3a4f991fe0df4a7adf9d7c4b3041d5776a13a2e3fba" Oct 29 23:00:05.545096 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:05.545071 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8_odh-ai-gateway-infra(c80b13fa-d61d-448a-9862-cd99080d88ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8" podUID="c80b13fa-d61d-448a-9862-cd99080d88ea" Oct 29 23:00:05.608696 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.608659 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istiod-ca-cert\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.608959 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.608936 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-workload-certs\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.609124 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.608987 4821 scope.go:122] "RemoveContainer" containerID="908d37e2ac37b9950f89a53d72856272812205bf14c6049094449410db57a3a1" Oct 29 23:00:05.609282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.609102 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-data\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.610187 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.610156 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-envoy\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.610291 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.610224 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-workload-socket\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.610483 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.610456 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmlff\" (UniqueName: \"kubernetes.io/projected/5bfcb45f-0e94-4e58-bc91-9157be0569ed-kube-api-access-wmlff\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.610552 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.610504 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-podinfo\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.610614 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.610558 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-credential-socket\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.610664 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.610618 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-token\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.678927 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.678879 4821 scope.go:122] "RemoveContainer" containerID="19da036f3264d37334120b91f5302185fc7bf04a57cd068d67843504062acdb7" Oct 29 23:00:05.696278 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.696139 4821 scope.go:122] "RemoveContainer" containerID="14771963bfa27aaa4cc258bbe51a71259dffa5cd7b883d9b50935dd54fac0f5c" Oct 29 23:00:05.712529 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.712495 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmlff\" (UniqueName: \"kubernetes.io/projected/5bfcb45f-0e94-4e58-bc91-9157be0569ed-kube-api-access-wmlff\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.712655 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.712544 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-podinfo\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.712655 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.712590 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-credential-socket\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.712655 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.712645 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-token\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.712846 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.712832 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istiod-ca-cert\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.713007 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.712922 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-workload-certs\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.713007 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.712966 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-data\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.713007 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.712995 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-envoy\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.713144 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.713023 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-workload-socket\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.713510 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.713489 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-workload-socket\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.715086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.714795 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-credential-socket\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.715086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.715035 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-data\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.715316 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.715298 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-workload-certs\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.716013 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.715744 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istiod-ca-cert\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.719190 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.718799 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-podinfo\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.719190 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.719128 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-envoy\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.741718 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.741686 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-token\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.742318 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.741921 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmlff\" (UniqueName: \"kubernetes.io/projected/5bfcb45f-0e94-4e58-bc91-9157be0569ed-kube-api-access-wmlff\") pod \"e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:05.820701 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.820615 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk"] Oct 29 23:00:05.831222 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.831176 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd525f-8654dc8bfd-s5fdk"] Oct 29 23:00:05.840399 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:05.840359 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:06.174778 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:06.174743 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdbff57a-f530-441a-b708-8e3d2c64ed13" path="/var/lib/kubelet/pods/bdbff57a-f530-441a-b708-8e3d2c64ed13/volumes" Oct 29 23:00:06.218132 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:06.218104 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd"] Oct 29 23:00:06.218482 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:00:06.218459 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bfcb45f_0e94_4e58_bc91_9157be0569ed.slice/crio-abcf65a14f426b1562ccdb59be9c63e45f7bb722fa4200415761190dccf3891f WatchSource:0}: Error finding container abcf65a14f426b1562ccdb59be9c63e45f7bb722fa4200415761190dccf3891f: Status 404 returned error can't find the container with id abcf65a14f426b1562ccdb59be9c63e45f7bb722fa4200415761190dccf3891f Oct 29 23:00:06.221386 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:06.221356 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:00:06.221478 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:06.221416 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:00:06.221478 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:06.221448 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:00:06.554269 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:06.554160 4821 scope.go:122] "RemoveContainer" containerID="9327b2daa78c7949cc356e905c7498129f17c6a680122268dc3f4f85644d2d65" Oct 29 23:00:06.554709 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:06.554625 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq_odh-ai-gateway-infra(f7557d94-c264-4d93-847e-3e0c823acf34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq" podUID="f7557d94-c264-4d93-847e-3e0c823acf34" Oct 29 23:00:06.556066 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:06.556042 4821 scope.go:122] "RemoveContainer" containerID="7350dfa0fc6acf675c2f6b3538d8088fb3cbc9acbe91237a0f85d24971656549" Oct 29 23:00:06.556476 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:06.556450 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4_odh-ai-gateway-infra(82e062dd-2e5b-436b-b494-76382054e140)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4" podUID="82e062dd-2e5b-436b-b494-76382054e140" Oct 29 23:00:06.559083 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:06.559059 4821 scope.go:122] "RemoveContainer" containerID="478afa57a478b85cacf9f3a4f991fe0df4a7adf9d7c4b3041d5776a13a2e3fba" Oct 29 23:00:06.559492 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:06.559467 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8_odh-ai-gateway-infra(c80b13fa-d61d-448a-9862-cd99080d88ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8" podUID="c80b13fa-d61d-448a-9862-cd99080d88ea" Oct 29 23:00:06.561163 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:06.561144 4821 scope.go:122] "RemoveContainer" containerID="c80c4cb214e6b7824b39d27cc5b4afa5e0d58fb401938b0f01c9b1ea8ae11702" Oct 29 23:00:06.561523 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:06.561495 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6_odh-ai-gateway-infra(63b44f7f-36f3-4f59-a45c-cd9af978b207)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6" podUID="63b44f7f-36f3-4f59-a45c-cd9af978b207" Oct 29 23:00:06.562719 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:06.562669 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7608-f16f60" event={"ID":"3dbba931-b65f-4e62-aed3-35759a78d6d2","Type":"ContainerStarted","Data":"2934ea359a3b27117116447bcf59ec19dab24367c4d797fb6953277ac6051e19"} Oct 29 23:00:06.564822 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:06.564795 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" event={"ID":"5bfcb45f-0e94-4e58-bc91-9157be0569ed","Type":"ContainerStarted","Data":"c7c4d2d9ebfd51c6aeb68c9d6aebaaccffae8ac0057e67f724ef1bb81fd13a00"} Oct 29 23:00:06.564926 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:06.564884 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" event={"ID":"5bfcb45f-0e94-4e58-bc91-9157be0569ed","Type":"ContainerStarted","Data":"abcf65a14f426b1562ccdb59be9c63e45f7bb722fa4200415761190dccf3891f"} Oct 29 23:00:06.567092 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:06.567073 4821 scope.go:122] "RemoveContainer" containerID="989050deb63aebb9a62b44233dba4253e5f573e46e63fd1cc6fe34eb0ee4f297" Oct 29 23:00:06.567345 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:06.567326 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph_odh-ai-gateway-infra(a4b1e79c-0035-4bae-bc76-94d40a09c66c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph" podUID="a4b1e79c-0035-4bae-bc76-94d40a09c66c" Oct 29 23:00:06.675233 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:06.675144 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7608-f16f60" podStartSLOduration=2.675124958 podStartE2EDuration="2.675124958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:06.667989992 +0000 UTC m=+1060.608537904" watchObservedRunningTime="2026-10-29 23:00:06.675124958 +0000 UTC m=+1060.615672869" Oct 29 23:00:06.794311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:06.794252 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" podStartSLOduration=1.7942364039999998 podStartE2EDuration="1.794236404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:06.788659346 +0000 UTC m=+1060.729207257" watchObservedRunningTime="2026-10-29 23:00:06.794236404 +0000 UTC m=+1060.734784313" Oct 29 23:00:06.841436 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:06.841350 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:07.581434 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:07.581390 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7608-f16f60"] Oct 29 23:00:07.618280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:07.618241 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7608-f16f60"] Oct 29 23:00:07.827923 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:07.827886 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx"] Oct 29 23:00:07.829026 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:07.829003 4821 cpu_manager.go:414] "containerMap: removing container" podUID="3dbba931-b65f-4e62-aed3-35759a78d6d2" containerName="test-curl-7608-f16f60" Oct 29 23:00:07.829074 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:07.829031 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dbba931-b65f-4e62-aed3-35759a78d6d2" containerName="test-curl-7608-f16f60" Oct 29 23:00:07.829285 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:07.829271 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dbba931-b65f-4e62-aed3-35759a78d6d2" containerName="test-curl-7608-f16f60" Oct 29 23:00:07.833830 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:07.833771 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx" Oct 29 23:00:07.852570 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:07.852523 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:07.856062 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:07.856036 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx"] Oct 29 23:00:07.856502 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:07.856474 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0791d5d8-74c9-49ef-a717-bfb3623d32ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx\" (UID: \"0791d5d8-74c9-49ef-a717-bfb3623d32ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx" Oct 29 23:00:07.856640 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:07.856622 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0791d5d8-74c9-49ef-a717-bfb3623d32ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx\" (UID: \"0791d5d8-74c9-49ef-a717-bfb3623d32ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx" Oct 29 23:00:07.856746 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:07.856729 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf678\" (UniqueName: \"kubernetes.io/projected/0791d5d8-74c9-49ef-a717-bfb3623d32ca-kube-api-access-qf678\") pod \"maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx\" (UID: \"0791d5d8-74c9-49ef-a717-bfb3623d32ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx" Oct 29 23:00:07.858507 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:07.858487 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:00:07.957988 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:07.957953 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0791d5d8-74c9-49ef-a717-bfb3623d32ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx\" (UID: \"0791d5d8-74c9-49ef-a717-bfb3623d32ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx" Oct 29 23:00:07.958172 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:07.958047 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0791d5d8-74c9-49ef-a717-bfb3623d32ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx\" (UID: \"0791d5d8-74c9-49ef-a717-bfb3623d32ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx" Oct 29 23:00:07.958172 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:07.958141 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf678\" (UniqueName: \"kubernetes.io/projected/0791d5d8-74c9-49ef-a717-bfb3623d32ca-kube-api-access-qf678\") pod \"maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx\" (UID: \"0791d5d8-74c9-49ef-a717-bfb3623d32ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx" Oct 29 23:00:07.958768 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:07.958745 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0791d5d8-74c9-49ef-a717-bfb3623d32ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx\" (UID: \"0791d5d8-74c9-49ef-a717-bfb3623d32ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx" Oct 29 23:00:07.960726 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:07.960699 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0791d5d8-74c9-49ef-a717-bfb3623d32ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx\" (UID: \"0791d5d8-74c9-49ef-a717-bfb3623d32ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx" Oct 29 23:00:07.981956 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:07.981900 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf678\" (UniqueName: \"kubernetes.io/projected/0791d5d8-74c9-49ef-a717-bfb3623d32ca-kube-api-access-qf678\") pod \"maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx\" (UID: \"0791d5d8-74c9-49ef-a717-bfb3623d32ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx" Oct 29 23:00:08.154553 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.154515 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx" Oct 29 23:00:08.464456 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:00:08.464425 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0791d5d8_74c9_49ef_a717_bfb3623d32ca.slice/crio-cd495d0e4bfefde6449148fac9a75c1d885b507acbe35f053194dc740df8bd13 WatchSource:0}: Error finding container cd495d0e4bfefde6449148fac9a75c1d885b507acbe35f053194dc740df8bd13: Status 404 returned error can't find the container with id cd495d0e4bfefde6449148fac9a75c1d885b507acbe35f053194dc740df8bd13 Oct 29 23:00:08.464684 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.464656 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx"] Oct 29 23:00:08.619736 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.619698 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx" event={"ID":"0791d5d8-74c9-49ef-a717-bfb3623d32ca","Type":"ContainerStarted","Data":"cd495d0e4bfefde6449148fac9a75c1d885b507acbe35f053194dc740df8bd13"} Oct 29 23:00:08.622034 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.622007 4821 generic.go:356] "Generic (PLEG): container finished" podID="525944e9-324d-4398-88bb-eeeea15478ce" containerID="6ee2a527c5334137142f8768f3796a398a0b7798d68eb0d02b21faa44f1b9341" exitCode=0 Oct 29 23:00:08.622246 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.622082 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" event={"ID":"525944e9-324d-4398-88bb-eeeea15478ce","Type":"ContainerDied","Data":"6ee2a527c5334137142f8768f3796a398a0b7798d68eb0d02b21faa44f1b9341"} Oct 29 23:00:08.622246 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.622217 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7608-f16f60" podUID="3dbba931-b65f-4e62-aed3-35759a78d6d2" containerName="test-curl-7608-f16f60" containerID="cri-o://2934ea359a3b27117116447bcf59ec19dab24367c4d797fb6953277ac6051e19" gracePeriod=2 Oct 29 23:00:08.622820 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.622729 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:08.624878 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.624854 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:00:08.772248 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.772213 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 23:00:08.874711 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.874677 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-workload-certs\") pod \"525944e9-324d-4398-88bb-eeeea15478ce\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " Oct 29 23:00:08.874884 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.874724 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-istio-data\") pod \"525944e9-324d-4398-88bb-eeeea15478ce\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " Oct 29 23:00:08.874884 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.874812 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/525944e9-324d-4398-88bb-eeeea15478ce-istio-token\" (UniqueName: \"kubernetes.io/projected/525944e9-324d-4398-88bb-eeeea15478ce-istio-token\") pod \"525944e9-324d-4398-88bb-eeeea15478ce\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " Oct 29 23:00:08.874884 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.874847 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-credential-socket\") pod \"525944e9-324d-4398-88bb-eeeea15478ce\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " Oct 29 23:00:08.875052 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.874892 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-workload-socket\") pod \"525944e9-324d-4398-88bb-eeeea15478ce\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " Oct 29 23:00:08.875052 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.874959 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/525944e9-324d-4398-88bb-eeeea15478ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/525944e9-324d-4398-88bb-eeeea15478ce-istiod-ca-cert\") pod \"525944e9-324d-4398-88bb-eeeea15478ce\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " Oct 29 23:00:08.875052 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.874955 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-workload-certs" pod "525944e9-324d-4398-88bb-eeeea15478ce" (UID: "525944e9-324d-4398-88bb-eeeea15478ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:08.875433 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.875359 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/525944e9-324d-4398-88bb-eeeea15478ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/525944e9-324d-4398-88bb-eeeea15478ce-istio-podinfo\") pod \"525944e9-324d-4398-88bb-eeeea15478ce\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " Oct 29 23:00:08.875433 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.875408 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-istio-envoy\") pod \"525944e9-324d-4398-88bb-eeeea15478ce\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " Oct 29 23:00:08.875598 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.875437 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/525944e9-324d-4398-88bb-eeeea15478ce-kube-api-access-jjbpr\" (UniqueName: \"kubernetes.io/projected/525944e9-324d-4398-88bb-eeeea15478ce-kube-api-access-jjbpr\") pod \"525944e9-324d-4398-88bb-eeeea15478ce\" (UID: \"525944e9-324d-4398-88bb-eeeea15478ce\") " Oct 29 23:00:08.875598 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.875538 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-istio-data" pod "525944e9-324d-4398-88bb-eeeea15478ce" (UID: "525944e9-324d-4398-88bb-eeeea15478ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:08.875753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.875731 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-credential-socket" pod "525944e9-324d-4398-88bb-eeeea15478ce" (UID: "525944e9-324d-4398-88bb-eeeea15478ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:08.875843 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.875823 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/525944e9-324d-4398-88bb-eeeea15478ce-istiod-ca-cert" pod "525944e9-324d-4398-88bb-eeeea15478ce" (UID: "525944e9-324d-4398-88bb-eeeea15478ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:08.876144 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.876118 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-workload-socket" pod "525944e9-324d-4398-88bb-eeeea15478ce" (UID: "525944e9-324d-4398-88bb-eeeea15478ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:08.876715 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.876692 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:08.876857 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.876844 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:08.876955 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.876940 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:08.877055 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.877042 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:08.877138 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.877127 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/525944e9-324d-4398-88bb-eeeea15478ce-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:08.878589 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.878431 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-istio-envoy" pod "525944e9-324d-4398-88bb-eeeea15478ce" (UID: "525944e9-324d-4398-88bb-eeeea15478ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:08.878589 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.878545 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525944e9-324d-4398-88bb-eeeea15478ce-kube-api-access-jjbpr" pod "525944e9-324d-4398-88bb-eeeea15478ce" (UID: "525944e9-324d-4398-88bb-eeeea15478ce"). InnerVolumeSpecName "kube-api-access-jjbpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:08.878773 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.878749 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525944e9-324d-4398-88bb-eeeea15478ce-istio-token" pod "525944e9-324d-4398-88bb-eeeea15478ce" (UID: "525944e9-324d-4398-88bb-eeeea15478ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:08.879777 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.879750 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/525944e9-324d-4398-88bb-eeeea15478ce-istio-podinfo" pod "525944e9-324d-4398-88bb-eeeea15478ce" (UID: "525944e9-324d-4398-88bb-eeeea15478ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:00:08.979093 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.979063 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/525944e9-324d-4398-88bb-eeeea15478ce-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:08.979093 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.979094 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/525944e9-324d-4398-88bb-eeeea15478ce-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:08.979274 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.979108 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/525944e9-324d-4398-88bb-eeeea15478ce-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:08.979274 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:08.979122 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjbpr\" (UniqueName: \"kubernetes.io/projected/525944e9-324d-4398-88bb-eeeea15478ce-kube-api-access-jjbpr\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:09.186059 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.186028 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n"] Oct 29 23:00:09.186749 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.186735 4821 cpu_manager.go:414] "containerMap: removing container" podUID="525944e9-324d-4398-88bb-eeeea15478ce" containerName="istio-proxy" Oct 29 23:00:09.186808 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.186750 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="525944e9-324d-4398-88bb-eeeea15478ce" containerName="istio-proxy" Oct 29 23:00:09.186859 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.186849 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="525944e9-324d-4398-88bb-eeeea15478ce" containerName="istio-proxy" Oct 29 23:00:09.191740 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.191724 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n" Oct 29 23:00:09.206947 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.206920 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d3e268-serving-cert\"" Oct 29 23:00:09.214308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.214278 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n"] Oct 29 23:00:09.283517 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.283477 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgqxf\" (UniqueName: \"kubernetes.io/projected/650729e2-7050-4990-b7f5-ec4fa4c51947-kube-api-access-kgqxf\") pod \"maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n\" (UID: \"650729e2-7050-4990-b7f5-ec4fa4c51947\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n" Oct 29 23:00:09.283704 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.283612 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/650729e2-7050-4990-b7f5-ec4fa4c51947-metrics-certs\") pod \"maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n\" (UID: \"650729e2-7050-4990-b7f5-ec4fa4c51947\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n" Oct 29 23:00:09.283704 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.283683 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/650729e2-7050-4990-b7f5-ec4fa4c51947-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n\" (UID: \"650729e2-7050-4990-b7f5-ec4fa4c51947\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n" Oct 29 23:00:09.283837 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.283720 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/650729e2-7050-4990-b7f5-ec4fa4c51947-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n\" (UID: \"650729e2-7050-4990-b7f5-ec4fa4c51947\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n" Oct 29 23:00:09.385763 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.385589 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgqxf\" (UniqueName: \"kubernetes.io/projected/650729e2-7050-4990-b7f5-ec4fa4c51947-kube-api-access-kgqxf\") pod \"maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n\" (UID: \"650729e2-7050-4990-b7f5-ec4fa4c51947\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n" Oct 29 23:00:09.386233 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.386158 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/650729e2-7050-4990-b7f5-ec4fa4c51947-metrics-certs\") pod \"maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n\" (UID: \"650729e2-7050-4990-b7f5-ec4fa4c51947\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n" Oct 29 23:00:09.386535 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.386485 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/650729e2-7050-4990-b7f5-ec4fa4c51947-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n\" (UID: \"650729e2-7050-4990-b7f5-ec4fa4c51947\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n" Oct 29 23:00:09.386850 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.386680 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/650729e2-7050-4990-b7f5-ec4fa4c51947-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n\" (UID: \"650729e2-7050-4990-b7f5-ec4fa4c51947\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n" Oct 29 23:00:09.395758 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.395728 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/650729e2-7050-4990-b7f5-ec4fa4c51947-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n\" (UID: \"650729e2-7050-4990-b7f5-ec4fa4c51947\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n" Oct 29 23:00:09.402640 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.402520 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/650729e2-7050-4990-b7f5-ec4fa4c51947-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n\" (UID: \"650729e2-7050-4990-b7f5-ec4fa4c51947\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n" Oct 29 23:00:09.403683 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.403496 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/650729e2-7050-4990-b7f5-ec4fa4c51947-metrics-certs\") pod \"maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n\" (UID: \"650729e2-7050-4990-b7f5-ec4fa4c51947\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n" Oct 29 23:00:09.417960 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.411899 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgqxf\" (UniqueName: \"kubernetes.io/projected/650729e2-7050-4990-b7f5-ec4fa4c51947-kube-api-access-kgqxf\") pod \"maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n\" (UID: \"650729e2-7050-4990-b7f5-ec4fa4c51947\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n" Oct 29 23:00:09.506390 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.506354 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n" Oct 29 23:00:09.640945 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.640902 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" event={"ID":"525944e9-324d-4398-88bb-eeeea15478ce","Type":"ContainerDied","Data":"68ed6d9868a98e0946a938294cd97c6a3435988aede105d9272b0ba6ae7ea823"} Oct 29 23:00:09.641425 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.640960 4821 scope.go:122] "RemoveContainer" containerID="6ee2a527c5334137142f8768f3796a398a0b7798d68eb0d02b21faa44f1b9341" Oct 29 23:00:09.641425 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.641084 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45" Oct 29 23:00:09.643695 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.643663 4821 generic.go:356] "Generic (PLEG): container finished" podID="0791d5d8-74c9-49ef-a717-bfb3623d32ca" containerID="e83be1af553a11aa74cd2f214051b2ddf8debf14c1216f25f7c72c5d2a682a19" exitCode=0 Oct 29 23:00:09.643814 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.643740 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx" event={"ID":"0791d5d8-74c9-49ef-a717-bfb3623d32ca","Type":"ContainerDied","Data":"e83be1af553a11aa74cd2f214051b2ddf8debf14c1216f25f7c72c5d2a682a19"} Oct 29 23:00:09.738083 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.738004 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45"] Oct 29 23:00:09.753228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.753169 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-27d16a-openshift-default-869755f997-p2h45"] Oct 29 23:00:09.832875 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:00:09.832841 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod650729e2_7050_4990_b7f5_ec4fa4c51947.slice/crio-310e71b37812f5f9e160c606702eaedff017aea546c93ec742a41946484e711a WatchSource:0}: Error finding container 310e71b37812f5f9e160c606702eaedff017aea546c93ec742a41946484e711a: Status 404 returned error can't find the container with id 310e71b37812f5f9e160c606702eaedff017aea546c93ec742a41946484e711a Oct 29 23:00:09.833382 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:09.833356 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n"] Oct 29 23:00:10.173709 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:10.173679 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="525944e9-324d-4398-88bb-eeeea15478ce" path="/var/lib/kubelet/pods/525944e9-324d-4398-88bb-eeeea15478ce/volumes" Oct 29 23:00:10.703305 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:10.699636 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n" event={"ID":"650729e2-7050-4990-b7f5-ec4fa4c51947","Type":"ContainerStarted","Data":"c43c9e058ffb72c1a0d7396903a814e8e81cb154f3940fb3dd0dd845c0435f5e"} Oct 29 23:00:10.703305 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:10.699715 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n" event={"ID":"650729e2-7050-4990-b7f5-ec4fa4c51947","Type":"ContainerStarted","Data":"310e71b37812f5f9e160c606702eaedff017aea546c93ec742a41946484e711a"} Oct 29 23:00:10.703305 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:10.699931 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n" Oct 29 23:00:10.770712 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:10.764083 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n" podStartSLOduration=1.449647621 podStartE2EDuration="1.764063863s" totalImagesPullingTime="314.416242ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:10.763398404 +0000 UTC m=+1064.703946316" watchObservedRunningTime="2026-10-29 23:00:10.764063863 +0000 UTC m=+1064.704611774" Oct 29 23:00:11.134727 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:11.134695 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-f16f60" Oct 29 23:00:11.139768 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:11.139745 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx" Oct 29 23:00:11.188395 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:11.187801 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0791d5d8-74c9-49ef-a717-bfb3623d32ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0791d5d8-74c9-49ef-a717-bfb3623d32ca-maas-api-xks-ca\") pod \"0791d5d8-74c9-49ef-a717-bfb3623d32ca\" (UID: \"0791d5d8-74c9-49ef-a717-bfb3623d32ca\") " Oct 29 23:00:11.188395 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:11.187909 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbba931-b65f-4e62-aed3-35759a78d6d2-kube-api-access-4vm4q\" (UniqueName: \"kubernetes.io/projected/3dbba931-b65f-4e62-aed3-35759a78d6d2-kube-api-access-4vm4q\") pod \"3dbba931-b65f-4e62-aed3-35759a78d6d2\" (UID: \"3dbba931-b65f-4e62-aed3-35759a78d6d2\") " Oct 29 23:00:11.188395 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:11.188061 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0791d5d8-74c9-49ef-a717-bfb3623d32ca-kube-api-access-qf678\" (UniqueName: \"kubernetes.io/projected/0791d5d8-74c9-49ef-a717-bfb3623d32ca-kube-api-access-qf678\") pod \"0791d5d8-74c9-49ef-a717-bfb3623d32ca\" (UID: \"0791d5d8-74c9-49ef-a717-bfb3623d32ca\") " Oct 29 23:00:11.188395 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:11.188391 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0791d5d8-74c9-49ef-a717-bfb3623d32ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0791d5d8-74c9-49ef-a717-bfb3623d32ca-maas-api-openshift-ca\") pod \"0791d5d8-74c9-49ef-a717-bfb3623d32ca\" (UID: \"0791d5d8-74c9-49ef-a717-bfb3623d32ca\") " Oct 29 23:00:11.191003 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:11.190971 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0791d5d8-74c9-49ef-a717-bfb3623d32ca-maas-api-openshift-ca" pod "0791d5d8-74c9-49ef-a717-bfb3623d32ca" (UID: "0791d5d8-74c9-49ef-a717-bfb3623d32ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:11.194656 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:11.194620 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0791d5d8-74c9-49ef-a717-bfb3623d32ca-kube-api-access-qf678" pod "0791d5d8-74c9-49ef-a717-bfb3623d32ca" (UID: "0791d5d8-74c9-49ef-a717-bfb3623d32ca"). InnerVolumeSpecName "kube-api-access-qf678". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:11.195946 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:11.195914 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0791d5d8-74c9-49ef-a717-bfb3623d32ca-maas-api-xks-ca" pod "0791d5d8-74c9-49ef-a717-bfb3623d32ca" (UID: "0791d5d8-74c9-49ef-a717-bfb3623d32ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:11.196773 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:11.196748 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbba931-b65f-4e62-aed3-35759a78d6d2-kube-api-access-4vm4q" pod "3dbba931-b65f-4e62-aed3-35759a78d6d2" (UID: "3dbba931-b65f-4e62-aed3-35759a78d6d2"). InnerVolumeSpecName "kube-api-access-4vm4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:11.291633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:11.291535 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf678\" (UniqueName: \"kubernetes.io/projected/0791d5d8-74c9-49ef-a717-bfb3623d32ca-kube-api-access-qf678\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:11.291633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:11.291576 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0791d5d8-74c9-49ef-a717-bfb3623d32ca-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:11.291633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:11.291592 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0791d5d8-74c9-49ef-a717-bfb3623d32ca-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:11.291633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:11.291606 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vm4q\" (UniqueName: \"kubernetes.io/projected/3dbba931-b65f-4e62-aed3-35759a78d6d2-kube-api-access-4vm4q\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:11.703218 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:11.703173 4821 generic.go:356] "Generic (PLEG): container finished" podID="3dbba931-b65f-4e62-aed3-35759a78d6d2" containerID="2934ea359a3b27117116447bcf59ec19dab24367c4d797fb6953277ac6051e19" exitCode=137 Oct 29 23:00:11.703385 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:11.703232 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7608-f16f60" Oct 29 23:00:11.703385 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:11.703246 4821 scope.go:122] "RemoveContainer" containerID="2934ea359a3b27117116447bcf59ec19dab24367c4d797fb6953277ac6051e19" Oct 29 23:00:11.705417 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:11.705395 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx" Oct 29 23:00:11.705564 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:11.705424 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx" event={"ID":"0791d5d8-74c9-49ef-a717-bfb3623d32ca","Type":"ContainerDied","Data":"cd495d0e4bfefde6449148fac9a75c1d885b507acbe35f053194dc740df8bd13"} Oct 29 23:00:11.705726 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:11.705592 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd495d0e4bfefde6449148fac9a75c1d885b507acbe35f053194dc740df8bd13" Oct 29 23:00:11.716779 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:11.716759 4821 scope.go:122] "RemoveContainer" containerID="2934ea359a3b27117116447bcf59ec19dab24367c4d797fb6953277ac6051e19" Oct 29 23:00:11.717156 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:11.717124 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2934ea359a3b27117116447bcf59ec19dab24367c4d797fb6953277ac6051e19\": container with ID starting with 2934ea359a3b27117116447bcf59ec19dab24367c4d797fb6953277ac6051e19 not found: ID does not exist" containerID="2934ea359a3b27117116447bcf59ec19dab24367c4d797fb6953277ac6051e19" Oct 29 23:00:11.717321 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:11.717167 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2934ea359a3b27117116447bcf59ec19dab24367c4d797fb6953277ac6051e19"} err="failed to get container status \"2934ea359a3b27117116447bcf59ec19dab24367c4d797fb6953277ac6051e19\": rpc error: code = NotFound desc = could not find container \"2934ea359a3b27117116447bcf59ec19dab24367c4d797fb6953277ac6051e19\": container with ID starting with 2934ea359a3b27117116447bcf59ec19dab24367c4d797fb6953277ac6051e19 not found: ID does not exist" Oct 29 23:00:11.730311 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:11.730259 4821 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7608-f16f60\" is forbidden: User \"system:node:ip-10-0-130-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-252.ec2.internal' and this object" podUID="3dbba931-b65f-4e62-aed3-35759a78d6d2" pod="opendatahub/test-curl-7608-f16f60" Oct 29 23:00:11.743243 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:11.743171 4821 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7608-f16f60\" is forbidden: User \"system:node:ip-10-0-130-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-252.ec2.internal' and this object" podUID="3dbba931-b65f-4e62-aed3-35759a78d6d2" pod="opendatahub/test-curl-7608-f16f60" Oct 29 23:00:11.965583 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:11.965489 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c"] Oct 29 23:00:11.965762 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:11.965737 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" podUID="bdee46ad-6d49-4415-8d35-060d32549a59" containerName="istio-proxy" containerID="cri-o://4818ca980de99bfbc746c1b8eff4e3bc4c963cf6ae6f657e0931c28d426f8901" gracePeriod=30 Oct 29 23:00:11.980547 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:11.980510 4821 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7608-f16f60\" is forbidden: User \"system:node:ip-10-0-130-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-252.ec2.internal' and this object" podUID="3dbba931-b65f-4e62-aed3-35759a78d6d2" pod="opendatahub/test-curl-7608-f16f60" Oct 29 23:00:12.175253 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:12.175186 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dbba931-b65f-4e62-aed3-35759a78d6d2" path="/var/lib/kubelet/pods/3dbba931-b65f-4e62-aed3-35759a78d6d2/volumes" Oct 29 23:00:16.168301 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:16.168269 4821 scope.go:122] "RemoveContainer" containerID="c80c4cb214e6b7824b39d27cc5b4afa5e0d58fb401938b0f01c9b1ea8ae11702" Oct 29 23:00:16.168794 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:16.168495 4821 scope.go:122] "RemoveContainer" containerID="9327b2daa78c7949cc356e905c7498129f17c6a680122268dc3f4f85644d2d65" Oct 29 23:00:16.168794 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:16.168648 4821 scope.go:122] "RemoveContainer" containerID="989050deb63aebb9a62b44233dba4253e5f573e46e63fd1cc6fe34eb0ee4f297" Oct 29 23:00:16.168794 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:16.168723 4821 scope.go:122] "RemoveContainer" containerID="1fc0fd4d2c5f75a5f750e4a678c02420deae6aea6c6a74c34236787cf8f62b05" Oct 29 23:00:16.169562 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:16.169281 4821 scope.go:122] "RemoveContainer" containerID="7350dfa0fc6acf675c2f6b3538d8088fb3cbc9acbe91237a0f85d24971656549" Oct 29 23:00:17.163640 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.163521 4821 scope.go:122] "RemoveContainer" containerID="3d5a53acc7f3563e87d2105cb3739f7fe710e385a4148821449389bce739a517" Oct 29 23:00:17.441286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.432988 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 23:00:17.581452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.554303 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdee46ad-6d49-4415-8d35-060d32549a59-kube-api-access-gpmt2\" (UniqueName: \"kubernetes.io/projected/bdee46ad-6d49-4415-8d35-060d32549a59-kube-api-access-gpmt2\") pod \"bdee46ad-6d49-4415-8d35-060d32549a59\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " Oct 29 23:00:17.581452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.554406 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-credential-socket\") pod \"bdee46ad-6d49-4415-8d35-060d32549a59\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " Oct 29 23:00:17.581452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.554463 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdee46ad-6d49-4415-8d35-060d32549a59-istio-token\" (UniqueName: \"kubernetes.io/projected/bdee46ad-6d49-4415-8d35-060d32549a59-istio-token\") pod \"bdee46ad-6d49-4415-8d35-060d32549a59\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " Oct 29 23:00:17.581452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.554530 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-workload-certs\") pod \"bdee46ad-6d49-4415-8d35-060d32549a59\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " Oct 29 23:00:17.581452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.554664 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-istio-data\") pod \"bdee46ad-6d49-4415-8d35-060d32549a59\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " Oct 29 23:00:17.581452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.554761 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdee46ad-6d49-4415-8d35-060d32549a59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdee46ad-6d49-4415-8d35-060d32549a59-istiod-ca-cert\") pod \"bdee46ad-6d49-4415-8d35-060d32549a59\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " Oct 29 23:00:17.581452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.554847 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-workload-socket\") pod \"bdee46ad-6d49-4415-8d35-060d32549a59\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " Oct 29 23:00:17.581452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.554928 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-istio-envoy\") pod \"bdee46ad-6d49-4415-8d35-060d32549a59\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " Oct 29 23:00:17.581452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.554974 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdee46ad-6d49-4415-8d35-060d32549a59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdee46ad-6d49-4415-8d35-060d32549a59-istio-podinfo\") pod \"bdee46ad-6d49-4415-8d35-060d32549a59\" (UID: \"bdee46ad-6d49-4415-8d35-060d32549a59\") " Oct 29 23:00:17.581452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.555490 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-istio-data" pod "bdee46ad-6d49-4415-8d35-060d32549a59" (UID: "bdee46ad-6d49-4415-8d35-060d32549a59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:17.581452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.555656 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-credential-socket" pod "bdee46ad-6d49-4415-8d35-060d32549a59" (UID: "bdee46ad-6d49-4415-8d35-060d32549a59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:17.581452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.556174 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdee46ad-6d49-4415-8d35-060d32549a59-istiod-ca-cert" pod "bdee46ad-6d49-4415-8d35-060d32549a59" (UID: "bdee46ad-6d49-4415-8d35-060d32549a59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:17.581452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.556405 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-workload-socket" pod "bdee46ad-6d49-4415-8d35-060d32549a59" (UID: "bdee46ad-6d49-4415-8d35-060d32549a59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:17.581452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.561464 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-workload-certs" pod "bdee46ad-6d49-4415-8d35-060d32549a59" (UID: "bdee46ad-6d49-4415-8d35-060d32549a59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:17.581452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.561818 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdee46ad-6d49-4415-8d35-060d32549a59-kube-api-access-gpmt2" pod "bdee46ad-6d49-4415-8d35-060d32549a59" (UID: "bdee46ad-6d49-4415-8d35-060d32549a59"). InnerVolumeSpecName "kube-api-access-gpmt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:17.581452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.562163 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpmt2\" (UniqueName: \"kubernetes.io/projected/bdee46ad-6d49-4415-8d35-060d32549a59-kube-api-access-gpmt2\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:17.581452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.562189 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:17.581452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.562229 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:17.581452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.562244 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:17.581452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.562258 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdee46ad-6d49-4415-8d35-060d32549a59-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:17.581452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.562271 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:17.669753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.634562 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdee46ad-6d49-4415-8d35-060d32549a59-istio-token" pod "bdee46ad-6d49-4415-8d35-060d32549a59" (UID: "bdee46ad-6d49-4415-8d35-060d32549a59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:17.669753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.669701 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdee46ad-6d49-4415-8d35-060d32549a59-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:17.753649 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.752395 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n" Oct 29 23:00:17.828149 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.818388 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdee46ad-6d49-4415-8d35-060d32549a59-istio-podinfo" pod "bdee46ad-6d49-4415-8d35-060d32549a59" (UID: "bdee46ad-6d49-4415-8d35-060d32549a59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:00:17.828149 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.819826 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-istio-envoy" pod "bdee46ad-6d49-4415-8d35-060d32549a59" (UID: "bdee46ad-6d49-4415-8d35-060d32549a59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:17.828149 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.824618 4821 generic.go:356] "Generic (PLEG): container finished" podID="63b44f7f-36f3-4f59-a45c-cd9af978b207" containerID="0799c6a915d3d03d97f2dae693afc706555877c53ab4460296cce08aabf7c28f" exitCode=60 Oct 29 23:00:17.828149 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.824744 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6" event={"ID":"63b44f7f-36f3-4f59-a45c-cd9af978b207","Type":"ContainerDied","Data":"0799c6a915d3d03d97f2dae693afc706555877c53ab4460296cce08aabf7c28f"} Oct 29 23:00:17.828149 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.824802 4821 scope.go:122] "RemoveContainer" containerID="c80c4cb214e6b7824b39d27cc5b4afa5e0d58fb401938b0f01c9b1ea8ae11702" Oct 29 23:00:17.828149 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.825392 4821 scope.go:122] "RemoveContainer" containerID="0799c6a915d3d03d97f2dae693afc706555877c53ab4460296cce08aabf7c28f" Oct 29 23:00:17.828149 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:17.825892 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6_odh-ai-gateway-infra(63b44f7f-36f3-4f59-a45c-cd9af978b207)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6" podUID="63b44f7f-36f3-4f59-a45c-cd9af978b207" Oct 29 23:00:17.839580 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.838410 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4" event={"ID":"82e062dd-2e5b-436b-b494-76382054e140","Type":"ContainerStarted","Data":"08cfe850474ee60262dc5118fb4dcb472ed618f4aefcef03d86ee28f641248a0"} Oct 29 23:00:17.850628 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.850590 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph" event={"ID":"a4b1e79c-0035-4bae-bc76-94d40a09c66c","Type":"ContainerStarted","Data":"08f0080698295bd3f8f57bb0cca39f8aae7c8354abf9ebeb7ab395ae908e5282"} Oct 29 23:00:17.858303 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.857683 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb" event={"ID":"15027c98-a748-4aab-b0f0-5d913bfd61ff","Type":"ContainerStarted","Data":"86e5b07c3209e39b29a7a0583ce216cf32b25c4f9dad64ec5274b91cc306689f"} Oct 29 23:00:17.858303 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.858142 4821 scope.go:122] "RemoveContainer" containerID="86e5b07c3209e39b29a7a0583ce216cf32b25c4f9dad64ec5274b91cc306689f" Oct 29 23:00:17.858894 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:17.858836 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888580-mrxrb_odh-ai-gateway-infra(15027c98-a748-4aab-b0f0-5d913bfd61ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb" podUID="15027c98-a748-4aab-b0f0-5d913bfd61ff" Oct 29 23:00:17.903796 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.903715 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdee46ad-6d49-4415-8d35-060d32549a59-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:17.904008 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.903990 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdee46ad-6d49-4415-8d35-060d32549a59-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:17.922232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.920480 4821 generic.go:356] "Generic (PLEG): container finished" podID="bdee46ad-6d49-4415-8d35-060d32549a59" containerID="4818ca980de99bfbc746c1b8eff4e3bc4c963cf6ae6f657e0931c28d426f8901" exitCode=0 Oct 29 23:00:17.960224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.922975 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" Oct 29 23:00:17.960224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.922997 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" event={"ID":"bdee46ad-6d49-4415-8d35-060d32549a59","Type":"ContainerDied","Data":"4818ca980de99bfbc746c1b8eff4e3bc4c963cf6ae6f657e0931c28d426f8901"} Oct 29 23:00:17.960224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.944922 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c" event={"ID":"bdee46ad-6d49-4415-8d35-060d32549a59","Type":"ContainerDied","Data":"3587c83a59559219c56236d8613373ac8cf2c2e1c958817f42127c0a0b90136c"} Oct 29 23:00:17.964406 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.964377 4821 scope.go:122] "RemoveContainer" containerID="4818ca980de99bfbc746c1b8eff4e3bc4c963cf6ae6f657e0931c28d426f8901" Oct 29 23:00:17.970418 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.969316 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq" event={"ID":"f7557d94-c264-4d93-847e-3e0c823acf34","Type":"ContainerStarted","Data":"0ba38d0fbd4c902155628cb87e5ccfdd6d3a6c05230fa85357715f68c5e8f16a"} Oct 29 23:00:17.970418 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:17.969818 4821 scope.go:122] "RemoveContainer" containerID="0ba38d0fbd4c902155628cb87e5ccfdd6d3a6c05230fa85357715f68c5e8f16a" Oct 29 23:00:17.971024 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:17.970669 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq_odh-ai-gateway-infra(f7557d94-c264-4d93-847e-3e0c823acf34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq" podUID="f7557d94-c264-4d93-847e-3e0c823acf34" Oct 29 23:00:18.050082 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.050049 4821 scope.go:122] "RemoveContainer" containerID="4818ca980de99bfbc746c1b8eff4e3bc4c963cf6ae6f657e0931c28d426f8901" Oct 29 23:00:18.061422 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:18.061383 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4818ca980de99bfbc746c1b8eff4e3bc4c963cf6ae6f657e0931c28d426f8901\": container with ID starting with 4818ca980de99bfbc746c1b8eff4e3bc4c963cf6ae6f657e0931c28d426f8901 not found: ID does not exist" containerID="4818ca980de99bfbc746c1b8eff4e3bc4c963cf6ae6f657e0931c28d426f8901" Oct 29 23:00:18.061806 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.061761 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4818ca980de99bfbc746c1b8eff4e3bc4c963cf6ae6f657e0931c28d426f8901"} err="failed to get container status \"4818ca980de99bfbc746c1b8eff4e3bc4c963cf6ae6f657e0931c28d426f8901\": rpc error: code = NotFound desc = could not find container \"4818ca980de99bfbc746c1b8eff4e3bc4c963cf6ae6f657e0931c28d426f8901\": container with ID starting with 4818ca980de99bfbc746c1b8eff4e3bc4c963cf6ae6f657e0931c28d426f8901 not found: ID does not exist" Oct 29 23:00:18.069799 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.068720 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj"] Oct 29 23:00:18.069799 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.069748 4821 cpu_manager.go:414] "containerMap: removing container" podUID="0791d5d8-74c9-49ef-a717-bfb3623d32ca" containerName="revoke-keys" Oct 29 23:00:18.069799 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.069768 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0791d5d8-74c9-49ef-a717-bfb3623d32ca" containerName="revoke-keys" Oct 29 23:00:18.070136 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.069821 4821 cpu_manager.go:414] "containerMap: removing container" podUID="bdee46ad-6d49-4415-8d35-060d32549a59" containerName="istio-proxy" Oct 29 23:00:18.070136 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.069829 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdee46ad-6d49-4415-8d35-060d32549a59" containerName="istio-proxy" Oct 29 23:00:18.070136 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.070021 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="0791d5d8-74c9-49ef-a717-bfb3623d32ca" containerName="revoke-keys" Oct 29 23:00:18.070136 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.070038 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdee46ad-6d49-4415-8d35-060d32549a59" containerName="istio-proxy" Oct 29 23:00:18.079562 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.079535 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.080661 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.080637 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj"] Oct 29 23:00:18.098728 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.094554 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-774bff-openshift-default-dockercfg-rw2r7\"" Oct 29 23:00:18.152642 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.152611 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c"] Oct 29 23:00:18.160289 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.160145 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-cd525f-openshift-default-5567b55479-vnr9c"] Oct 29 23:00:18.187254 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.186482 4821 scope.go:122] "RemoveContainer" containerID="478afa57a478b85cacf9f3a4f991fe0df4a7adf9d7c4b3041d5776a13a2e3fba" Oct 29 23:00:18.241174 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.240359 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdee46ad-6d49-4415-8d35-060d32549a59" path="/var/lib/kubelet/pods/bdee46ad-6d49-4415-8d35-060d32549a59/volumes" Oct 29 23:00:18.247277 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.247188 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-envoy\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.247462 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.247443 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-credential-socket\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.247612 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.247594 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istiod-ca-cert\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.247742 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.247728 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb48p\" (UniqueName: \"kubernetes.io/projected/59c2911d-7186-44a7-9b7b-e5c55f1c185a-kube-api-access-pb48p\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.248174 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.248152 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-workload-socket\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.248360 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.248316 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-workload-certs\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.248466 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.248443 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-data\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.248793 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.248579 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-token\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.248793 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.248668 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-podinfo\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.364273 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.363761 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-workload-socket\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.364849 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.364513 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-workload-certs\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.365085 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.364249 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-workload-socket\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.365085 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.364816 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-workload-certs\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.365715 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.365694 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-data\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.366557 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.366437 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-data\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.367141 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.367122 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-token\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.368326 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.367635 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-podinfo\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.368524 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.368504 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-envoy\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.369010 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.368990 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-credential-socket\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.370432 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.370292 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-credential-socket\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.370943 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.370922 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istiod-ca-cert\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.371887 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.371738 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istiod-ca-cert\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.372067 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.372047 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb48p\" (UniqueName: \"kubernetes.io/projected/59c2911d-7186-44a7-9b7b-e5c55f1c185a-kube-api-access-pb48p\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.417429 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.417334 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb48p\" (UniqueName: \"kubernetes.io/projected/59c2911d-7186-44a7-9b7b-e5c55f1c185a-kube-api-access-pb48p\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.417673 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.417486 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-envoy\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.417966 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.417944 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-podinfo\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.420904 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.420637 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-token\") pod \"e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.445738 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.445704 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:18.840734 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.840698 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj"] Oct 29 23:00:18.841812 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:00:18.841777 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c2911d_7186_44a7_9b7b_e5c55f1c185a.slice/crio-24a62afd074ba78643ea9bc1e73218c12d17b4321d22dcfa3e1b49871a0adafb WatchSource:0}: Error finding container 24a62afd074ba78643ea9bc1e73218c12d17b4321d22dcfa3e1b49871a0adafb: Status 404 returned error can't find the container with id 24a62afd074ba78643ea9bc1e73218c12d17b4321d22dcfa3e1b49871a0adafb Oct 29 23:00:18.864945 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.864893 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:00:18.865079 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.864993 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:00:18.865079 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.865036 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:00:18.896235 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.892845 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6"] Oct 29 23:00:18.938344 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.938249 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4"] Oct 29 23:00:18.995634 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:18.988924 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph"] Oct 29 23:00:19.010459 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.010429 4821 generic.go:356] "Generic (PLEG): container finished" podID="15027c98-a748-4aab-b0f0-5d913bfd61ff" containerID="86e5b07c3209e39b29a7a0583ce216cf32b25c4f9dad64ec5274b91cc306689f" exitCode=60 Oct 29 23:00:19.010711 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.010690 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb" event={"ID":"15027c98-a748-4aab-b0f0-5d913bfd61ff","Type":"ContainerDied","Data":"86e5b07c3209e39b29a7a0583ce216cf32b25c4f9dad64ec5274b91cc306689f"} Oct 29 23:00:19.010865 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.010849 4821 scope.go:122] "RemoveContainer" containerID="1fc0fd4d2c5f75a5f750e4a678c02420deae6aea6c6a74c34236787cf8f62b05" Oct 29 23:00:19.012023 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.011601 4821 scope.go:122] "RemoveContainer" containerID="86e5b07c3209e39b29a7a0583ce216cf32b25c4f9dad64ec5274b91cc306689f" Oct 29 23:00:19.021280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.021223 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb"] Oct 29 23:00:19.021678 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:19.021652 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888580-mrxrb_odh-ai-gateway-infra(15027c98-a748-4aab-b0f0-5d913bfd61ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb" podUID="15027c98-a748-4aab-b0f0-5d913bfd61ff" Oct 29 23:00:19.025986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.025957 4821 generic.go:356] "Generic (PLEG): container finished" podID="f7557d94-c264-4d93-847e-3e0c823acf34" containerID="0ba38d0fbd4c902155628cb87e5ccfdd6d3a6c05230fa85357715f68c5e8f16a" exitCode=60 Oct 29 23:00:19.027241 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.027217 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq" event={"ID":"f7557d94-c264-4d93-847e-3e0c823acf34","Type":"ContainerDied","Data":"0ba38d0fbd4c902155628cb87e5ccfdd6d3a6c05230fa85357715f68c5e8f16a"} Oct 29 23:00:19.027855 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.027835 4821 scope.go:122] "RemoveContainer" containerID="0ba38d0fbd4c902155628cb87e5ccfdd6d3a6c05230fa85357715f68c5e8f16a" Oct 29 23:00:19.028306 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:19.028282 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq_odh-ai-gateway-infra(f7557d94-c264-4d93-847e-3e0c823acf34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq" podUID="f7557d94-c264-4d93-847e-3e0c823acf34" Oct 29 23:00:19.030726 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.030705 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" event={"ID":"59c2911d-7186-44a7-9b7b-e5c55f1c185a","Type":"ContainerStarted","Data":"24a62afd074ba78643ea9bc1e73218c12d17b4321d22dcfa3e1b49871a0adafb"} Oct 29 23:00:19.033172 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.033156 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8" event={"ID":"c80b13fa-d61d-448a-9862-cd99080d88ea","Type":"ContainerStarted","Data":"8b67fd6f46f5b15a9c3a17c5c4eefa1b776edbd13e9ffe8825772dabd22f8948"} Oct 29 23:00:19.040347 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.040320 4821 generic.go:356] "Generic (PLEG): container finished" podID="82e062dd-2e5b-436b-b494-76382054e140" containerID="08cfe850474ee60262dc5118fb4dcb472ed618f4aefcef03d86ee28f641248a0" exitCode=60 Oct 29 23:00:19.040542 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.040524 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4" event={"ID":"82e062dd-2e5b-436b-b494-76382054e140","Type":"ContainerDied","Data":"08cfe850474ee60262dc5118fb4dcb472ed618f4aefcef03d86ee28f641248a0"} Oct 29 23:00:19.044123 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.044101 4821 generic.go:356] "Generic (PLEG): container finished" podID="a4b1e79c-0035-4bae-bc76-94d40a09c66c" containerID="08f0080698295bd3f8f57bb0cca39f8aae7c8354abf9ebeb7ab395ae908e5282" exitCode=60 Oct 29 23:00:19.044445 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.044426 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph" event={"ID":"a4b1e79c-0035-4bae-bc76-94d40a09c66c","Type":"ContainerDied","Data":"08f0080698295bd3f8f57bb0cca39f8aae7c8354abf9ebeb7ab395ae908e5282"} Oct 29 23:00:19.048117 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.048100 4821 generic.go:356] "Generic (PLEG): container finished" podID="13a1ca2f-190b-42d5-8ead-3d64c06111d2" containerID="09b2609782849f5ad2234cc866260a65a90a75cdd7add30bdf72f2645f67a4dd" exitCode=60 Oct 29 23:00:19.048325 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.048303 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9" event={"ID":"13a1ca2f-190b-42d5-8ead-3d64c06111d2","Type":"ContainerDied","Data":"09b2609782849f5ad2234cc866260a65a90a75cdd7add30bdf72f2645f67a4dd"} Oct 29 23:00:19.048757 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.048741 4821 scope.go:122] "RemoveContainer" containerID="09b2609782849f5ad2234cc866260a65a90a75cdd7add30bdf72f2645f67a4dd" Oct 29 23:00:19.049109 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:19.049087 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9_odh-ai-gateway-infra(13a1ca2f-190b-42d5-8ead-3d64c06111d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9" podUID="13a1ca2f-190b-42d5-8ead-3d64c06111d2" Oct 29 23:00:19.074646 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.073993 4821 scope.go:122] "RemoveContainer" containerID="9327b2daa78c7949cc356e905c7498129f17c6a680122268dc3f4f85644d2d65" Oct 29 23:00:19.152349 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.152313 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq"] Oct 29 23:00:19.309293 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.305592 4821 scope.go:122] "RemoveContainer" containerID="7350dfa0fc6acf675c2f6b3538d8088fb3cbc9acbe91237a0f85d24971656549" Oct 29 23:00:19.467568 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.443752 4821 scope.go:122] "RemoveContainer" containerID="989050deb63aebb9a62b44233dba4253e5f573e46e63fd1cc6fe34eb0ee4f297" Oct 29 23:00:19.467568 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.445882 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:19.467568 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.450658 4821 patch_prober.go:29] interesting pod/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 29 23:00:19.467568 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.450712 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" podUID="59c2911d-7186-44a7-9b7b-e5c55f1c185a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 29 23:00:19.516626 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.515992 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" podStartSLOduration=1.5159722979999999 podStartE2EDuration="1.515972298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:19.441492854 +0000 UTC m=+1073.382040767" watchObservedRunningTime="2026-10-29 23:00:19.515972298 +0000 UTC m=+1073.456520208" Oct 29 23:00:19.661369 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.657375 4821 scope.go:122] "RemoveContainer" containerID="3d5a53acc7f3563e87d2105cb3739f7fe710e385a4148821449389bce739a517" Oct 29 23:00:19.835956 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.835496 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph" Oct 29 23:00:19.964279 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.948488 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4b1e79c-0035-4bae-bc76-94d40a09c66c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b1e79c-0035-4bae-bc76-94d40a09c66c-maas-api-service-ca\") pod \"a4b1e79c-0035-4bae-bc76-94d40a09c66c\" (UID: \"a4b1e79c-0035-4bae-bc76-94d40a09c66c\") " Oct 29 23:00:19.970157 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:19.970004 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4b1e79c-0035-4bae-bc76-94d40a09c66c-kube-api-access-v5w2d\" (UniqueName: \"kubernetes.io/projected/a4b1e79c-0035-4bae-bc76-94d40a09c66c-kube-api-access-v5w2d\") pod \"a4b1e79c-0035-4bae-bc76-94d40a09c66c\" (UID: \"a4b1e79c-0035-4bae-bc76-94d40a09c66c\") " Oct 29 23:00:20.044309 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.031012 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b1e79c-0035-4bae-bc76-94d40a09c66c-kube-api-access-v5w2d" pod "a4b1e79c-0035-4bae-bc76-94d40a09c66c" (UID: "a4b1e79c-0035-4bae-bc76-94d40a09c66c"). InnerVolumeSpecName "kube-api-access-v5w2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:20.044309 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.041018 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b1e79c-0035-4bae-bc76-94d40a09c66c-maas-api-service-ca" pod "a4b1e79c-0035-4bae-bc76-94d40a09c66c" (UID: "a4b1e79c-0035-4bae-bc76-94d40a09c66c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:20.092282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.073751 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b1e79c-0035-4bae-bc76-94d40a09c66c-maas-api-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.092282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.073801 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5w2d\" (UniqueName: \"kubernetes.io/projected/a4b1e79c-0035-4bae-bc76-94d40a09c66c-kube-api-access-v5w2d\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.128221 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.127978 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4" Oct 29 23:00:20.135381 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.135351 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6" Oct 29 23:00:20.136755 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.136730 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9"] Oct 29 23:00:20.138594 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.138569 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4" Oct 29 23:00:20.138913 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.138566 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4" event={"ID":"82e062dd-2e5b-436b-b494-76382054e140","Type":"ContainerDied","Data":"a1857047143719d44fc97920a51d626225e072e8967b3a4c8c6404c3b844ea7e"} Oct 29 23:00:20.139302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.139290 4821 scope.go:122] "RemoveContainer" containerID="08cfe850474ee60262dc5118fb4dcb472ed618f4aefcef03d86ee28f641248a0" Oct 29 23:00:20.140252 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.140183 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph" Oct 29 23:00:20.140645 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.140620 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph" event={"ID":"a4b1e79c-0035-4bae-bc76-94d40a09c66c","Type":"ContainerDied","Data":"f09321ad8f5e68ac1157342e1b7cfc391bfc00b51a7ac889893f2469d3dda8f9"} Oct 29 23:00:20.159094 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.158999 4821 scope.go:122] "RemoveContainer" containerID="08f0080698295bd3f8f57bb0cca39f8aae7c8354abf9ebeb7ab395ae908e5282" Oct 29 23:00:20.165286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.161471 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" event={"ID":"59c2911d-7186-44a7-9b7b-e5c55f1c185a","Type":"ContainerStarted","Data":"9cd89555453f6592bcd7e640bbbe6c16d378d123539af7cda8219930f3338018"} Oct 29 23:00:20.191753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.171138 4821 generic.go:356] "Generic (PLEG): container finished" podID="c80b13fa-d61d-448a-9862-cd99080d88ea" containerID="8b67fd6f46f5b15a9c3a17c5c4eefa1b776edbd13e9ffe8825772dabd22f8948" exitCode=60 Oct 29 23:00:20.191753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.174769 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82e062dd-2e5b-436b-b494-76382054e140-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82e062dd-2e5b-436b-b494-76382054e140-maas-api-service-ca\") pod \"82e062dd-2e5b-436b-b494-76382054e140\" (UID: \"82e062dd-2e5b-436b-b494-76382054e140\") " Oct 29 23:00:20.191753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.174864 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63b44f7f-36f3-4f59-a45c-cd9af978b207-kube-api-access-h4br9\" (UniqueName: \"kubernetes.io/projected/63b44f7f-36f3-4f59-a45c-cd9af978b207-kube-api-access-h4br9\") pod \"63b44f7f-36f3-4f59-a45c-cd9af978b207\" (UID: \"63b44f7f-36f3-4f59-a45c-cd9af978b207\") " Oct 29 23:00:20.191753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.174894 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63b44f7f-36f3-4f59-a45c-cd9af978b207-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b44f7f-36f3-4f59-a45c-cd9af978b207-maas-api-service-ca\") pod \"63b44f7f-36f3-4f59-a45c-cd9af978b207\" (UID: \"63b44f7f-36f3-4f59-a45c-cd9af978b207\") " Oct 29 23:00:20.191753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.174931 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e062dd-2e5b-436b-b494-76382054e140-kube-api-access-wc62z\" (UniqueName: \"kubernetes.io/projected/82e062dd-2e5b-436b-b494-76382054e140-kube-api-access-wc62z\") pod \"82e062dd-2e5b-436b-b494-76382054e140\" (UID: \"82e062dd-2e5b-436b-b494-76382054e140\") " Oct 29 23:00:20.191753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.185318 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e062dd-2e5b-436b-b494-76382054e140-kube-api-access-wc62z" pod "82e062dd-2e5b-436b-b494-76382054e140" (UID: "82e062dd-2e5b-436b-b494-76382054e140"). InnerVolumeSpecName "kube-api-access-wc62z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:20.191753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.185654 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63b44f7f-36f3-4f59-a45c-cd9af978b207-maas-api-service-ca" pod "63b44f7f-36f3-4f59-a45c-cd9af978b207" (UID: "63b44f7f-36f3-4f59-a45c-cd9af978b207"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:20.191753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.185971 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e062dd-2e5b-436b-b494-76382054e140-maas-api-service-ca" pod "82e062dd-2e5b-436b-b494-76382054e140" (UID: "82e062dd-2e5b-436b-b494-76382054e140"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:20.191753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.186601 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b44f7f-36f3-4f59-a45c-cd9af978b207-kube-api-access-h4br9" pod "63b44f7f-36f3-4f59-a45c-cd9af978b207" (UID: "63b44f7f-36f3-4f59-a45c-cd9af978b207"). InnerVolumeSpecName "kube-api-access-h4br9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:20.209590 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.209540 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8"] Oct 29 23:00:20.209849 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.209820 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8" event={"ID":"c80b13fa-d61d-448a-9862-cd99080d88ea","Type":"ContainerDied","Data":"8b67fd6f46f5b15a9c3a17c5c4eefa1b776edbd13e9ffe8825772dabd22f8948"} Oct 29 23:00:20.226005 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.225972 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6" event={"ID":"63b44f7f-36f3-4f59-a45c-cd9af978b207","Type":"ContainerDied","Data":"7ee84f7bb957d4ad0a3712a4b23bac7048a13de981a4856e568e0c731838a886"} Oct 29 23:00:20.226568 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.226326 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6" Oct 29 23:00:20.285612 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.261815 4821 scope.go:122] "RemoveContainer" containerID="478afa57a478b85cacf9f3a4f991fe0df4a7adf9d7c4b3041d5776a13a2e3fba" Oct 29 23:00:20.285612 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.278179 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82e062dd-2e5b-436b-b494-76382054e140-maas-api-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.285612 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.278230 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4br9\" (UniqueName: \"kubernetes.io/projected/63b44f7f-36f3-4f59-a45c-cd9af978b207-kube-api-access-h4br9\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.285612 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.278244 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b44f7f-36f3-4f59-a45c-cd9af978b207-maas-api-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.285612 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.278257 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc62z\" (UniqueName: \"kubernetes.io/projected/82e062dd-2e5b-436b-b494-76382054e140-kube-api-access-wc62z\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.320617 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.320249 4821 scope.go:122] "RemoveContainer" containerID="0799c6a915d3d03d97f2dae693afc706555877c53ab4460296cce08aabf7c28f" Oct 29 23:00:20.440248 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.433862 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph"] Oct 29 23:00:20.454365 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.453777 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a1d1da-29888580-8cfph"] Oct 29 23:00:20.455703 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.455499 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:20.529428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.529400 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb" Oct 29 23:00:20.572629 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.572520 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6"] Oct 29 23:00:20.581034 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.580954 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9" Oct 29 23:00:20.605552 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.602728 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15027c98-a748-4aab-b0f0-5d913bfd61ff-kube-api-access-wcn8z\" (UniqueName: \"kubernetes.io/projected/15027c98-a748-4aab-b0f0-5d913bfd61ff-kube-api-access-wcn8z\") pod \"15027c98-a748-4aab-b0f0-5d913bfd61ff\" (UID: \"15027c98-a748-4aab-b0f0-5d913bfd61ff\") " Oct 29 23:00:20.605552 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.603169 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15027c98-a748-4aab-b0f0-5d913bfd61ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15027c98-a748-4aab-b0f0-5d913bfd61ff-maas-api-service-ca\") pod \"15027c98-a748-4aab-b0f0-5d913bfd61ff\" (UID: \"15027c98-a748-4aab-b0f0-5d913bfd61ff\") " Oct 29 23:00:20.606238 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.606067 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15027c98-a748-4aab-b0f0-5d913bfd61ff-maas-api-service-ca" pod "15027c98-a748-4aab-b0f0-5d913bfd61ff" (UID: "15027c98-a748-4aab-b0f0-5d913bfd61ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:20.612100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.612015 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15027c98-a748-4aab-b0f0-5d913bfd61ff-kube-api-access-wcn8z" pod "15027c98-a748-4aab-b0f0-5d913bfd61ff" (UID: "15027c98-a748-4aab-b0f0-5d913bfd61ff"). InnerVolumeSpecName "kube-api-access-wcn8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:20.612737 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.612307 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-49dbb5-29888580-xttt6"] Oct 29 23:00:20.631648 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.628251 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8" Oct 29 23:00:20.651967 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.651940 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq" Oct 29 23:00:20.660281 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.660194 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4"] Oct 29 23:00:20.667649 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.667514 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888580-lljz4"] Oct 29 23:00:20.705316 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.704985 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13a1ca2f-190b-42d5-8ead-3d64c06111d2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a1ca2f-190b-42d5-8ead-3d64c06111d2-maas-api-service-ca\") pod \"13a1ca2f-190b-42d5-8ead-3d64c06111d2\" (UID: \"13a1ca2f-190b-42d5-8ead-3d64c06111d2\") " Oct 29 23:00:20.705316 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.705042 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c80b13fa-d61d-448a-9862-cd99080d88ea-kube-api-access-6v4cs\" (UniqueName: \"kubernetes.io/projected/c80b13fa-d61d-448a-9862-cd99080d88ea-kube-api-access-6v4cs\") pod \"c80b13fa-d61d-448a-9862-cd99080d88ea\" (UID: \"c80b13fa-d61d-448a-9862-cd99080d88ea\") " Oct 29 23:00:20.705316 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.705071 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c80b13fa-d61d-448a-9862-cd99080d88ea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c80b13fa-d61d-448a-9862-cd99080d88ea-maas-api-service-ca\") pod \"c80b13fa-d61d-448a-9862-cd99080d88ea\" (UID: \"c80b13fa-d61d-448a-9862-cd99080d88ea\") " Oct 29 23:00:20.705316 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.705111 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13a1ca2f-190b-42d5-8ead-3d64c06111d2-kube-api-access-nm9vg\" (UniqueName: \"kubernetes.io/projected/13a1ca2f-190b-42d5-8ead-3d64c06111d2-kube-api-access-nm9vg\") pod \"13a1ca2f-190b-42d5-8ead-3d64c06111d2\" (UID: \"13a1ca2f-190b-42d5-8ead-3d64c06111d2\") " Oct 29 23:00:20.705674 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.705483 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7557d94-c264-4d93-847e-3e0c823acf34-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7557d94-c264-4d93-847e-3e0c823acf34-maas-api-service-ca\") pod \"f7557d94-c264-4d93-847e-3e0c823acf34\" (UID: \"f7557d94-c264-4d93-847e-3e0c823acf34\") " Oct 29 23:00:20.705674 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.705576 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7557d94-c264-4d93-847e-3e0c823acf34-kube-api-access-zfgcw\" (UniqueName: \"kubernetes.io/projected/f7557d94-c264-4d93-847e-3e0c823acf34-kube-api-access-zfgcw\") pod \"f7557d94-c264-4d93-847e-3e0c823acf34\" (UID: \"f7557d94-c264-4d93-847e-3e0c823acf34\") " Oct 29 23:00:20.708270 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.706389 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15027c98-a748-4aab-b0f0-5d913bfd61ff-maas-api-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.708270 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.706417 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcn8z\" (UniqueName: \"kubernetes.io/projected/15027c98-a748-4aab-b0f0-5d913bfd61ff-kube-api-access-wcn8z\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.708270 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.706472 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a1ca2f-190b-42d5-8ead-3d64c06111d2-maas-api-service-ca" pod "13a1ca2f-190b-42d5-8ead-3d64c06111d2" (UID: "13a1ca2f-190b-42d5-8ead-3d64c06111d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:20.708270 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.706814 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7557d94-c264-4d93-847e-3e0c823acf34-maas-api-service-ca" pod "f7557d94-c264-4d93-847e-3e0c823acf34" (UID: "f7557d94-c264-4d93-847e-3e0c823acf34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:20.708270 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.707125 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c80b13fa-d61d-448a-9862-cd99080d88ea-maas-api-service-ca" pod "c80b13fa-d61d-448a-9862-cd99080d88ea" (UID: "c80b13fa-d61d-448a-9862-cd99080d88ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:20.716426 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.716368 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a1ca2f-190b-42d5-8ead-3d64c06111d2-kube-api-access-nm9vg" pod "13a1ca2f-190b-42d5-8ead-3d64c06111d2" (UID: "13a1ca2f-190b-42d5-8ead-3d64c06111d2"). InnerVolumeSpecName "kube-api-access-nm9vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:20.717704 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.717675 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80b13fa-d61d-448a-9862-cd99080d88ea-kube-api-access-6v4cs" pod "c80b13fa-d61d-448a-9862-cd99080d88ea" (UID: "c80b13fa-d61d-448a-9862-cd99080d88ea"). InnerVolumeSpecName "kube-api-access-6v4cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:20.718042 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.718006 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7557d94-c264-4d93-847e-3e0c823acf34-kube-api-access-zfgcw" pod "f7557d94-c264-4d93-847e-3e0c823acf34" (UID: "f7557d94-c264-4d93-847e-3e0c823acf34"). InnerVolumeSpecName "kube-api-access-zfgcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:20.807569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.807539 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7557d94-c264-4d93-847e-3e0c823acf34-maas-api-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.807569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.807565 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfgcw\" (UniqueName: \"kubernetes.io/projected/f7557d94-c264-4d93-847e-3e0c823acf34-kube-api-access-zfgcw\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.807569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.807577 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a1ca2f-190b-42d5-8ead-3d64c06111d2-maas-api-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.807863 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.807587 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v4cs\" (UniqueName: \"kubernetes.io/projected/c80b13fa-d61d-448a-9862-cd99080d88ea-kube-api-access-6v4cs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.807863 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.807596 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c80b13fa-d61d-448a-9862-cd99080d88ea-maas-api-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.807863 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:20.807605 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm9vg\" (UniqueName: \"kubernetes.io/projected/13a1ca2f-190b-42d5-8ead-3d64c06111d2-kube-api-access-nm9vg\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:21.235714 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:21.235681 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8" Oct 29 23:00:21.235917 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:21.235688 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8" event={"ID":"c80b13fa-d61d-448a-9862-cd99080d88ea","Type":"ContainerDied","Data":"19387cac20947a83deeafec8de30edb45a2f10ee41069c402ebee6a2084ced58"} Oct 29 23:00:21.235917 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:21.235863 4821 scope.go:122] "RemoveContainer" containerID="8b67fd6f46f5b15a9c3a17c5c4eefa1b776edbd13e9ffe8825772dabd22f8948" Oct 29 23:00:21.241282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:21.241261 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9" Oct 29 23:00:21.241419 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:21.241272 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9" event={"ID":"13a1ca2f-190b-42d5-8ead-3d64c06111d2","Type":"ContainerDied","Data":"66cdc22bfa1f2189008738029ac7d1386f31cabb07c5a7cd71b8aba4b7afd4ae"} Oct 29 23:00:21.242958 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:21.242937 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb" Oct 29 23:00:21.243087 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:21.242958 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb" event={"ID":"15027c98-a748-4aab-b0f0-5d913bfd61ff","Type":"ContainerDied","Data":"8a2d554e564b8d413201ce5a0ce4293bb0dc4349af2898f74b2794bb1006c0a9"} Oct 29 23:00:21.244874 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:21.244749 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq" event={"ID":"f7557d94-c264-4d93-847e-3e0c823acf34","Type":"ContainerDied","Data":"b2ca1009266a5ded4d1bc67a84f7b70ef0df5f1e7b6b7ffb63a972c51eeef28f"} Oct 29 23:00:21.244874 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:21.244827 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:21.245641 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:21.245314 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq" Oct 29 23:00:21.246380 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:21.246081 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:00:21.268003 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:21.267982 4821 scope.go:122] "RemoveContainer" containerID="09b2609782849f5ad2234cc866260a65a90a75cdd7add30bdf72f2645f67a4dd" Oct 29 23:00:21.278285 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:21.278265 4821 scope.go:122] "RemoveContainer" containerID="86e5b07c3209e39b29a7a0583ce216cf32b25c4f9dad64ec5274b91cc306689f" Oct 29 23:00:21.288603 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:21.288444 4821 scope.go:122] "RemoveContainer" containerID="0ba38d0fbd4c902155628cb87e5ccfdd6d3a6c05230fa85357715f68c5e8f16a" Oct 29 23:00:21.377684 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:21.377648 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb"] Oct 29 23:00:21.384008 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:21.383980 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888580-mrxrb"] Oct 29 23:00:21.427496 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:21.427465 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9"] Oct 29 23:00:21.431947 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:21.431921 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e131af-29888580-ck5j9"] Oct 29 23:00:21.456923 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:21.456894 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8"] Oct 29 23:00:21.461725 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:21.461703 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a0dbf5-29888580-vvgc8"] Oct 29 23:00:21.493896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:21.493813 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq"] Oct 29 23:00:21.499689 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:21.499664 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36befd-29888580-cmrfq"] Oct 29 23:00:22.192831 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:22.192234 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13a1ca2f-190b-42d5-8ead-3d64c06111d2" path="/var/lib/kubelet/pods/13a1ca2f-190b-42d5-8ead-3d64c06111d2/volumes" Oct 29 23:00:22.193365 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:22.192866 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15027c98-a748-4aab-b0f0-5d913bfd61ff" path="/var/lib/kubelet/pods/15027c98-a748-4aab-b0f0-5d913bfd61ff/volumes" Oct 29 23:00:22.193578 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:22.193443 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63b44f7f-36f3-4f59-a45c-cd9af978b207" path="/var/lib/kubelet/pods/63b44f7f-36f3-4f59-a45c-cd9af978b207/volumes" Oct 29 23:00:22.194479 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:22.194458 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82e062dd-2e5b-436b-b494-76382054e140" path="/var/lib/kubelet/pods/82e062dd-2e5b-436b-b494-76382054e140/volumes" Oct 29 23:00:22.195007 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:22.194990 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4b1e79c-0035-4bae-bc76-94d40a09c66c" path="/var/lib/kubelet/pods/a4b1e79c-0035-4bae-bc76-94d40a09c66c/volumes" Oct 29 23:00:22.195527 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:22.195504 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c80b13fa-d61d-448a-9862-cd99080d88ea" path="/var/lib/kubelet/pods/c80b13fa-d61d-448a-9862-cd99080d88ea/volumes" Oct 29 23:00:22.196288 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:22.196269 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7557d94-c264-4d93-847e-3e0c823acf34" path="/var/lib/kubelet/pods/f7557d94-c264-4d93-847e-3e0c823acf34/volumes" Oct 29 23:00:23.730708 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.730674 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-774bff-b8475597b-gkstk"] Oct 29 23:00:23.731349 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731333 4821 cpu_manager.go:414] "containerMap: removing container" podUID="15027c98-a748-4aab-b0f0-5d913bfd61ff" containerName="cleanup" Oct 29 23:00:23.731418 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731350 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15027c98-a748-4aab-b0f0-5d913bfd61ff" containerName="cleanup" Oct 29 23:00:23.731418 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731369 4821 cpu_manager.go:414] "containerMap: removing container" podUID="f7557d94-c264-4d93-847e-3e0c823acf34" containerName="cleanup" Oct 29 23:00:23.731418 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731374 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7557d94-c264-4d93-847e-3e0c823acf34" containerName="cleanup" Oct 29 23:00:23.731418 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731379 4821 cpu_manager.go:414] "containerMap: removing container" podUID="c80b13fa-d61d-448a-9862-cd99080d88ea" containerName="cleanup" Oct 29 23:00:23.731418 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731384 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c80b13fa-d61d-448a-9862-cd99080d88ea" containerName="cleanup" Oct 29 23:00:23.731418 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731395 4821 cpu_manager.go:414] "containerMap: removing container" podUID="63b44f7f-36f3-4f59-a45c-cd9af978b207" containerName="cleanup" Oct 29 23:00:23.731418 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731399 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63b44f7f-36f3-4f59-a45c-cd9af978b207" containerName="cleanup" Oct 29 23:00:23.731418 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731406 4821 cpu_manager.go:414] "containerMap: removing container" podUID="f7557d94-c264-4d93-847e-3e0c823acf34" containerName="cleanup" Oct 29 23:00:23.731418 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731411 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7557d94-c264-4d93-847e-3e0c823acf34" containerName="cleanup" Oct 29 23:00:23.731418 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731421 4821 cpu_manager.go:414] "containerMap: removing container" podUID="82e062dd-2e5b-436b-b494-76382054e140" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731426 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e062dd-2e5b-436b-b494-76382054e140" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731436 4821 cpu_manager.go:414] "containerMap: removing container" podUID="82e062dd-2e5b-436b-b494-76382054e140" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731441 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e062dd-2e5b-436b-b494-76382054e140" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731448 4821 cpu_manager.go:414] "containerMap: removing container" podUID="a4b1e79c-0035-4bae-bc76-94d40a09c66c" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731452 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4b1e79c-0035-4bae-bc76-94d40a09c66c" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731458 4821 cpu_manager.go:414] "containerMap: removing container" podUID="13a1ca2f-190b-42d5-8ead-3d64c06111d2" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731463 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13a1ca2f-190b-42d5-8ead-3d64c06111d2" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731472 4821 cpu_manager.go:414] "containerMap: removing container" podUID="a4b1e79c-0035-4bae-bc76-94d40a09c66c" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731476 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4b1e79c-0035-4bae-bc76-94d40a09c66c" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731481 4821 cpu_manager.go:414] "containerMap: removing container" podUID="13a1ca2f-190b-42d5-8ead-3d64c06111d2" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731485 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13a1ca2f-190b-42d5-8ead-3d64c06111d2" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731499 4821 cpu_manager.go:414] "containerMap: removing container" podUID="15027c98-a748-4aab-b0f0-5d913bfd61ff" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731503 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15027c98-a748-4aab-b0f0-5d913bfd61ff" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731519 4821 cpu_manager.go:414] "containerMap: removing container" podUID="63b44f7f-36f3-4f59-a45c-cd9af978b207" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731523 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63b44f7f-36f3-4f59-a45c-cd9af978b207" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731536 4821 cpu_manager.go:414] "containerMap: removing container" podUID="c80b13fa-d61d-448a-9862-cd99080d88ea" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731542 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c80b13fa-d61d-448a-9862-cd99080d88ea" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731549 4821 cpu_manager.go:414] "containerMap: removing container" podUID="82e062dd-2e5b-436b-b494-76382054e140" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731553 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e062dd-2e5b-436b-b494-76382054e140" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731639 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="c80b13fa-d61d-448a-9862-cd99080d88ea" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731645 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="c80b13fa-d61d-448a-9862-cd99080d88ea" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731652 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="15027c98-a748-4aab-b0f0-5d913bfd61ff" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731658 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e062dd-2e5b-436b-b494-76382054e140" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731664 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="15027c98-a748-4aab-b0f0-5d913bfd61ff" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731672 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7557d94-c264-4d93-847e-3e0c823acf34" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731677 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="c80b13fa-d61d-448a-9862-cd99080d88ea" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731682 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="63b44f7f-36f3-4f59-a45c-cd9af978b207" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731687 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7557d94-c264-4d93-847e-3e0c823acf34" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731693 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e062dd-2e5b-436b-b494-76382054e140" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731700 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e062dd-2e5b-436b-b494-76382054e140" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731706 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4b1e79c-0035-4bae-bc76-94d40a09c66c" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731710 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="13a1ca2f-190b-42d5-8ead-3d64c06111d2" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731715 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="13a1ca2f-190b-42d5-8ead-3d64c06111d2" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731723 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="63b44f7f-36f3-4f59-a45c-cd9af978b207" containerName="cleanup" Oct 29 23:00:23.731722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.731728 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4b1e79c-0035-4bae-bc76-94d40a09c66c" containerName="cleanup" Oct 29 23:00:23.736850 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.736831 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-774bff-b8475597b-gkstk" Oct 29 23:00:23.759036 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.759012 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-774bff-b8475597b-gkstk"] Oct 29 23:00:23.772365 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.772260 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-774bff\"" Oct 29 23:00:23.772632 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.772610 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-774bff-dockercfg-kl8t7\"" Oct 29 23:00:23.882223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.865895 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789e0bd6-1808-413e-acc1-72537e22d107-plugins-config-volume\") pod \"payload-processing-e2e-api-b-774bff-b8475597b-gkstk\" (UID: \"789e0bd6-1808-413e-acc1-72537e22d107\") " pod="openshift-ingress/payload-processing-e2e-api-b-774bff-b8475597b-gkstk" Oct 29 23:00:23.892914 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.892870 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv55z\" (UniqueName: \"kubernetes.io/projected/789e0bd6-1808-413e-acc1-72537e22d107-kube-api-access-rv55z\") pod \"payload-processing-e2e-api-b-774bff-b8475597b-gkstk\" (UID: \"789e0bd6-1808-413e-acc1-72537e22d107\") " pod="openshift-ingress/payload-processing-e2e-api-b-774bff-b8475597b-gkstk" Oct 29 23:00:24.010276 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.999149 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv55z\" (UniqueName: \"kubernetes.io/projected/789e0bd6-1808-413e-acc1-72537e22d107-kube-api-access-rv55z\") pod \"payload-processing-e2e-api-b-774bff-b8475597b-gkstk\" (UID: \"789e0bd6-1808-413e-acc1-72537e22d107\") " pod="openshift-ingress/payload-processing-e2e-api-b-774bff-b8475597b-gkstk" Oct 29 23:00:24.010276 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:23.999233 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789e0bd6-1808-413e-acc1-72537e22d107-plugins-config-volume\") pod \"payload-processing-e2e-api-b-774bff-b8475597b-gkstk\" (UID: \"789e0bd6-1808-413e-acc1-72537e22d107\") " pod="openshift-ingress/payload-processing-e2e-api-b-774bff-b8475597b-gkstk" Oct 29 23:00:24.010276 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:24.000315 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789e0bd6-1808-413e-acc1-72537e22d107-plugins-config-volume\") pod \"payload-processing-e2e-api-b-774bff-b8475597b-gkstk\" (UID: \"789e0bd6-1808-413e-acc1-72537e22d107\") " pod="openshift-ingress/payload-processing-e2e-api-b-774bff-b8475597b-gkstk" Oct 29 23:00:24.033417 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:24.032665 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:00:24.033417 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:24.033349 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv55z\" (UniqueName: \"kubernetes.io/projected/789e0bd6-1808-413e-acc1-72537e22d107-kube-api-access-rv55z\") pod \"payload-processing-e2e-api-b-774bff-b8475597b-gkstk\" (UID: \"789e0bd6-1808-413e-acc1-72537e22d107\") " pod="openshift-ingress/payload-processing-e2e-api-b-774bff-b8475597b-gkstk" Oct 29 23:00:24.051860 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:24.051832 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-774bff-b8475597b-gkstk" Oct 29 23:00:24.409372 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:24.409332 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-774bff-b8475597b-gkstk"] Oct 29 23:00:24.410057 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:00:24.410027 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789e0bd6_1808_413e_acc1_72537e22d107.slice/crio-37730255a0f599bbc84302986cc4c79bd1143d927ddaf4cb57778577b5b6c88c WatchSource:0}: Error finding container 37730255a0f599bbc84302986cc4c79bd1143d927ddaf4cb57778577b5b6c88c: Status 404 returned error can't find the container with id 37730255a0f599bbc84302986cc4c79bd1143d927ddaf4cb57778577b5b6c88c Oct 29 23:00:25.302796 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:25.302757 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-774bff-b8475597b-gkstk" event={"ID":"789e0bd6-1808-413e-acc1-72537e22d107","Type":"ContainerStarted","Data":"b6d1b65506a6fcc33dee7fbcac489d6dac2ab0a5ddfdf1659458995e5b72a438"} Oct 29 23:00:25.303257 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:25.302990 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-774bff-b8475597b-gkstk" Oct 29 23:00:25.303257 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:25.303031 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-774bff-b8475597b-gkstk" event={"ID":"789e0bd6-1808-413e-acc1-72537e22d107","Type":"ContainerStarted","Data":"37730255a0f599bbc84302986cc4c79bd1143d927ddaf4cb57778577b5b6c88c"} Oct 29 23:00:25.345630 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:25.345567 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-774bff-b8475597b-gkstk" podStartSLOduration=2.345550392 podStartE2EDuration="2.345550392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:25.342582842 +0000 UTC m=+1079.283130758" watchObservedRunningTime="2026-10-29 23:00:25.345550392 +0000 UTC m=+1079.286098312" Oct 29 23:00:29.298896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:29.298862 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx"] Oct 29 23:00:29.309888 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:29.309856 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f3092a4d-q2xtx"] Oct 29 23:00:30.171970 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:30.171937 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0791d5d8-74c9-49ef-a717-bfb3623d32ca" path="/var/lib/kubelet/pods/0791d5d8-74c9-49ef-a717-bfb3623d32ca/volumes" Oct 29 23:00:34.763363 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:34.763332 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd"] Oct 29 23:00:35.119568 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:35.119530 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p"] Oct 29 23:00:35.119826 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:35.119796 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" podUID="f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" containerName="istio-proxy" containerID="cri-o://e92edf67746ce7c833ab122bb9f7b193f9f64c176eebe63410eb34fc22549e8e" gracePeriod=30 Oct 29 23:00:36.314869 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:36.314842 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-774bff-b8475597b-gkstk" Oct 29 23:00:38.031550 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:38.031515 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f42f78-cfknt"] Oct 29 23:00:38.042887 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:38.042855 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f42f78-cfknt"] Oct 29 23:00:38.172359 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:38.172322 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58366efd-cd68-466c-bc65-154613ed0ffe" path="/var/lib/kubelet/pods/58366efd-cd68-466c-bc65-154613ed0ffe/volumes" Oct 29 23:00:39.832959 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:39.832919 4821 patch_prober.go:29] interesting pod/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:00:39.833451 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:39.832984 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" podUID="f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:00:40.479748 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.479705 4821 generic.go:356] "Generic (PLEG): container finished" podID="f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" containerID="e92edf67746ce7c833ab122bb9f7b193f9f64c176eebe63410eb34fc22549e8e" exitCode=0 Oct 29 23:00:40.482459 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.479961 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" event={"ID":"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c","Type":"ContainerDied","Data":"e92edf67746ce7c833ab122bb9f7b193f9f64c176eebe63410eb34fc22549e8e"} Oct 29 23:00:40.482459 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.480021 4821 scope.go:122] "RemoveContainer" containerID="6f6549fdf04cd4065afd1a2034a9a1e24a16a6407e164ab8900d24fbe7b9cd18" Oct 29 23:00:40.504936 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.504908 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 23:00:40.612672 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.612642 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-workload-certs\") pod \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " Oct 29 23:00:40.612880 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.612756 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-credential-socket\") pod \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " Oct 29 23:00:40.612880 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.612810 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-data\") pod \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " Oct 29 23:00:40.612880 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.612877 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-token\" (UniqueName: \"kubernetes.io/projected/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-token\") pod \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " Oct 29 23:00:40.613057 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.612903 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-workload-socket\") pod \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " Oct 29 23:00:40.613057 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.612960 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istiod-ca-cert\") pod \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " Oct 29 23:00:40.613057 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.613001 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-kube-api-access-xvw79\" (UniqueName: \"kubernetes.io/projected/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-kube-api-access-xvw79\") pod \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " Oct 29 23:00:40.613057 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.613000 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-workload-certs" pod "f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" (UID: "f138dd09-a52b-4a74-bb6d-ffd90ca84f5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:40.613057 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.613023 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-podinfo\") pod \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " Oct 29 23:00:40.613057 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.613049 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-envoy\") pod \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\" (UID: \"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c\") " Oct 29 23:00:40.638106 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.637365 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-credential-socket" pod "f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" (UID: "f138dd09-a52b-4a74-bb6d-ffd90ca84f5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:40.638106 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.637671 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-data" pod "f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" (UID: "f138dd09-a52b-4a74-bb6d-ffd90ca84f5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:40.639327 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.639295 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:40.639461 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.639340 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:40.639461 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.639356 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:40.642696 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.642629 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-workload-socket" pod "f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" (UID: "f138dd09-a52b-4a74-bb6d-ffd90ca84f5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:40.643788 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.643749 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istiod-ca-cert" pod "f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" (UID: "f138dd09-a52b-4a74-bb6d-ffd90ca84f5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:40.643788 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.643745 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-token" pod "f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" (UID: "f138dd09-a52b-4a74-bb6d-ffd90ca84f5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:40.663280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.663220 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-kube-api-access-xvw79" pod "f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" (UID: "f138dd09-a52b-4a74-bb6d-ffd90ca84f5c"). InnerVolumeSpecName "kube-api-access-xvw79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:40.663438 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.663341 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-envoy" pod "f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" (UID: "f138dd09-a52b-4a74-bb6d-ffd90ca84f5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:40.663787 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.663758 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-podinfo" pod "f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" (UID: "f138dd09-a52b-4a74-bb6d-ffd90ca84f5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:00:40.741057 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.740963 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:40.741057 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.740996 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:40.741057 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.741007 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:40.741057 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.741037 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvw79\" (UniqueName: \"kubernetes.io/projected/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-kube-api-access-xvw79\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:40.741057 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.741047 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:40.741057 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:40.741056 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:41.489991 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:41.489898 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" event={"ID":"f138dd09-a52b-4a74-bb6d-ffd90ca84f5c","Type":"ContainerDied","Data":"4e864f61ad4a69839f2e50f5c76025d31cecb696dab1f4627b1bd30b9a7acb43"} Oct 29 23:00:41.489991 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:41.489968 4821 scope.go:122] "RemoveContainer" containerID="e92edf67746ce7c833ab122bb9f7b193f9f64c176eebe63410eb34fc22549e8e" Oct 29 23:00:41.490594 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:41.489908 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p" Oct 29 23:00:41.594905 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:41.594867 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p"] Oct 29 23:00:41.602428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:41.602392 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a0dbf5-openshift-default-777f74f5f9-k8l9p"] Oct 29 23:00:42.172813 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:42.172770 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" path="/var/lib/kubelet/pods/f138dd09-a52b-4a74-bb6d-ffd90ca84f5c/volumes" Oct 29 23:00:49.785546 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:49.785502 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" podUID="cb05c692-f4a5-49bf-a84d-ca3f03edd407" containerName="main" containerID="cri-o://6981b78dd69c1f162f07facba501c4fd64cd57ebbb19b4713df9502995776f48" gracePeriod=45 Oct 29 23:00:50.067312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.067277 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 23:00:50.111811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.111747 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-home\" (UniqueName: \"kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-home\") pod \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\" (UID: \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\") " Oct 29 23:00:50.111811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.111803 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-tmp-dir\") pod \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\" (UID: \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\") " Oct 29 23:00:50.112256 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.111878 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-dshm\") pod \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\" (UID: \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\") " Oct 29 23:00:50.112256 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.111910 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-model-cache\") pod \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\" (UID: \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\") " Oct 29 23:00:50.112256 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.111996 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb05c692-f4a5-49bf-a84d-ca3f03edd407-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb05c692-f4a5-49bf-a84d-ca3f03edd407-tls-certs\") pod \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\" (UID: \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\") " Oct 29 23:00:50.112256 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.112079 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb05c692-f4a5-49bf-a84d-ca3f03edd407-kube-api-access-bq52z\" (UniqueName: \"kubernetes.io/projected/cb05c692-f4a5-49bf-a84d-ca3f03edd407-kube-api-access-bq52z\") pod \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\" (UID: \"cb05c692-f4a5-49bf-a84d-ca3f03edd407\") " Oct 29 23:00:50.112571 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.112543 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-model-cache" pod "cb05c692-f4a5-49bf-a84d-ca3f03edd407" (UID: "cb05c692-f4a5-49bf-a84d-ca3f03edd407"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:50.113014 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.112986 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-home" pod "cb05c692-f4a5-49bf-a84d-ca3f03edd407" (UID: "cb05c692-f4a5-49bf-a84d-ca3f03edd407"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:50.113411 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.113042 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-tmp-dir" pod "cb05c692-f4a5-49bf-a84d-ca3f03edd407" (UID: "cb05c692-f4a5-49bf-a84d-ca3f03edd407"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:50.113411 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.113078 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:50.115084 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.115056 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb05c692-f4a5-49bf-a84d-ca3f03edd407-kube-api-access-bq52z" pod "cb05c692-f4a5-49bf-a84d-ca3f03edd407" (UID: "cb05c692-f4a5-49bf-a84d-ca3f03edd407"). InnerVolumeSpecName "kube-api-access-bq52z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:50.115669 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.115645 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-dshm" pod "cb05c692-f4a5-49bf-a84d-ca3f03edd407" (UID: "cb05c692-f4a5-49bf-a84d-ca3f03edd407"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:00:50.115669 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.115653 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb05c692-f4a5-49bf-a84d-ca3f03edd407-tls-certs" pod "cb05c692-f4a5-49bf-a84d-ca3f03edd407" (UID: "cb05c692-f4a5-49bf-a84d-ca3f03edd407"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:50.214936 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.214907 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:50.214936 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.214930 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb05c692-f4a5-49bf-a84d-ca3f03edd407-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:50.214936 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.214940 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq52z\" (UniqueName: \"kubernetes.io/projected/cb05c692-f4a5-49bf-a84d-ca3f03edd407-kube-api-access-bq52z\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:50.215245 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.214948 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:50.215245 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.214979 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb05c692-f4a5-49bf-a84d-ca3f03edd407-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:50.579903 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.579868 4821 generic.go:356] "Generic (PLEG): container finished" podID="cb05c692-f4a5-49bf-a84d-ca3f03edd407" containerID="6981b78dd69c1f162f07facba501c4fd64cd57ebbb19b4713df9502995776f48" exitCode=0 Oct 29 23:00:50.580090 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.579905 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" event={"ID":"cb05c692-f4a5-49bf-a84d-ca3f03edd407","Type":"ContainerDied","Data":"6981b78dd69c1f162f07facba501c4fd64cd57ebbb19b4713df9502995776f48"} Oct 29 23:00:50.580090 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.579922 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" Oct 29 23:00:50.580090 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.579957 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd" event={"ID":"cb05c692-f4a5-49bf-a84d-ca3f03edd407","Type":"ContainerDied","Data":"9ff4be40bf66fcfc42ad77102bac67f0f887bbcbf8342614fbe7569f36a99ab0"} Oct 29 23:00:50.580090 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.579976 4821 scope.go:122] "RemoveContainer" containerID="6981b78dd69c1f162f07facba501c4fd64cd57ebbb19b4713df9502995776f48" Oct 29 23:00:50.594407 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.594258 4821 scope.go:122] "RemoveContainer" containerID="6981b78dd69c1f162f07facba501c4fd64cd57ebbb19b4713df9502995776f48" Oct 29 23:00:50.594695 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:50.594670 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6981b78dd69c1f162f07facba501c4fd64cd57ebbb19b4713df9502995776f48\": container with ID starting with 6981b78dd69c1f162f07facba501c4fd64cd57ebbb19b4713df9502995776f48 not found: ID does not exist" containerID="6981b78dd69c1f162f07facba501c4fd64cd57ebbb19b4713df9502995776f48" Oct 29 23:00:50.594795 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.594705 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6981b78dd69c1f162f07facba501c4fd64cd57ebbb19b4713df9502995776f48"} err="failed to get container status \"6981b78dd69c1f162f07facba501c4fd64cd57ebbb19b4713df9502995776f48\": rpc error: code = NotFound desc = could not find container \"6981b78dd69c1f162f07facba501c4fd64cd57ebbb19b4713df9502995776f48\": container with ID starting with 6981b78dd69c1f162f07facba501c4fd64cd57ebbb19b4713df9502995776f48 not found: ID does not exist" Oct 29 23:00:50.632428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.632396 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd"] Oct 29 23:00:50.643860 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:50.643835 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-f3092a4d/e2e-lifecycle-model-f3092a4d-kserve-5d99847bf7-rvmzd"] Oct 29 23:00:51.596773 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.596739 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt"] Oct 29 23:00:51.597716 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.597690 4821 cpu_manager.go:414] "containerMap: removing container" podUID="cb05c692-f4a5-49bf-a84d-ca3f03edd407" containerName="main" Oct 29 23:00:51.597829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.597718 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb05c692-f4a5-49bf-a84d-ca3f03edd407" containerName="main" Oct 29 23:00:51.597829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.597746 4821 cpu_manager.go:414] "containerMap: removing container" podUID="c80b13fa-d61d-448a-9862-cd99080d88ea" containerName="cleanup" Oct 29 23:00:51.597829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.597753 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c80b13fa-d61d-448a-9862-cd99080d88ea" containerName="cleanup" Oct 29 23:00:51.597829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.597773 4821 cpu_manager.go:414] "containerMap: removing container" podUID="63b44f7f-36f3-4f59-a45c-cd9af978b207" containerName="cleanup" Oct 29 23:00:51.597829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.597781 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63b44f7f-36f3-4f59-a45c-cd9af978b207" containerName="cleanup" Oct 29 23:00:51.597829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.597792 4821 cpu_manager.go:414] "containerMap: removing container" podUID="f7557d94-c264-4d93-847e-3e0c823acf34" containerName="cleanup" Oct 29 23:00:51.597829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.597799 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7557d94-c264-4d93-847e-3e0c823acf34" containerName="cleanup" Oct 29 23:00:51.597829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.597822 4821 cpu_manager.go:414] "containerMap: removing container" podUID="f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" containerName="istio-proxy" Oct 29 23:00:51.597829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.597829 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" containerName="istio-proxy" Oct 29 23:00:51.598308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.597854 4821 cpu_manager.go:414] "containerMap: removing container" podUID="a4b1e79c-0035-4bae-bc76-94d40a09c66c" containerName="cleanup" Oct 29 23:00:51.598308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.597861 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4b1e79c-0035-4bae-bc76-94d40a09c66c" containerName="cleanup" Oct 29 23:00:51.598308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.597871 4821 cpu_manager.go:414] "containerMap: removing container" podUID="13a1ca2f-190b-42d5-8ead-3d64c06111d2" containerName="cleanup" Oct 29 23:00:51.598308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.597878 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13a1ca2f-190b-42d5-8ead-3d64c06111d2" containerName="cleanup" Oct 29 23:00:51.598308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.597902 4821 cpu_manager.go:414] "containerMap: removing container" podUID="f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" containerName="istio-proxy" Oct 29 23:00:51.598308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.597909 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" containerName="istio-proxy" Oct 29 23:00:51.598308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.597918 4821 cpu_manager.go:414] "containerMap: removing container" podUID="15027c98-a748-4aab-b0f0-5d913bfd61ff" containerName="cleanup" Oct 29 23:00:51.598308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.597925 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15027c98-a748-4aab-b0f0-5d913bfd61ff" containerName="cleanup" Oct 29 23:00:51.598308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.598094 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7557d94-c264-4d93-847e-3e0c823acf34" containerName="cleanup" Oct 29 23:00:51.598308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.598110 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" containerName="istio-proxy" Oct 29 23:00:51.598308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.598120 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4b1e79c-0035-4bae-bc76-94d40a09c66c" containerName="cleanup" Oct 29 23:00:51.598308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.598133 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="13a1ca2f-190b-42d5-8ead-3d64c06111d2" containerName="cleanup" Oct 29 23:00:51.598308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.598145 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="f138dd09-a52b-4a74-bb6d-ffd90ca84f5c" containerName="istio-proxy" Oct 29 23:00:51.598308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.598156 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="63b44f7f-36f3-4f59-a45c-cd9af978b207" containerName="cleanup" Oct 29 23:00:51.598308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.598169 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="15027c98-a748-4aab-b0f0-5d913bfd61ff" containerName="cleanup" Oct 29 23:00:51.598308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.598179 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb05c692-f4a5-49bf-a84d-ca3f03edd407" containerName="main" Oct 29 23:00:51.603668 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.603646 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt" Oct 29 23:00:51.635536 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.635490 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07038c7b-89e8-4830-949b-a0755eec2c3a-metrics-certs\") pod \"maas-api-e2e-api-b-774bff-f779d4d66-2tmwt\" (UID: \"07038c7b-89e8-4830-949b-a0755eec2c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt" Oct 29 23:00:51.635720 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.635547 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z7rq\" (UniqueName: \"kubernetes.io/projected/07038c7b-89e8-4830-949b-a0755eec2c3a-kube-api-access-7z7rq\") pod \"maas-api-e2e-api-b-774bff-f779d4d66-2tmwt\" (UID: \"07038c7b-89e8-4830-949b-a0755eec2c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt" Oct 29 23:00:51.635720 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.635616 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07038c7b-89e8-4830-949b-a0755eec2c3a-maas-api-tls\") pod \"maas-api-e2e-api-b-774bff-f779d4d66-2tmwt\" (UID: \"07038c7b-89e8-4830-949b-a0755eec2c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt" Oct 29 23:00:51.635844 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.635791 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07038c7b-89e8-4830-949b-a0755eec2c3a-openshift-service-ca\") pod \"maas-api-e2e-api-b-774bff-f779d4d66-2tmwt\" (UID: \"07038c7b-89e8-4830-949b-a0755eec2c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt" Oct 29 23:00:51.643115 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.643086 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-774bff-serving-cert\"" Oct 29 23:00:51.646427 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.646398 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt"] Oct 29 23:00:51.737136 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.737094 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07038c7b-89e8-4830-949b-a0755eec2c3a-metrics-certs\") pod \"maas-api-e2e-api-b-774bff-f779d4d66-2tmwt\" (UID: \"07038c7b-89e8-4830-949b-a0755eec2c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt" Oct 29 23:00:51.737136 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.737135 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z7rq\" (UniqueName: \"kubernetes.io/projected/07038c7b-89e8-4830-949b-a0755eec2c3a-kube-api-access-7z7rq\") pod \"maas-api-e2e-api-b-774bff-f779d4d66-2tmwt\" (UID: \"07038c7b-89e8-4830-949b-a0755eec2c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt" Oct 29 23:00:51.737433 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.737176 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07038c7b-89e8-4830-949b-a0755eec2c3a-maas-api-tls\") pod \"maas-api-e2e-api-b-774bff-f779d4d66-2tmwt\" (UID: \"07038c7b-89e8-4830-949b-a0755eec2c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt" Oct 29 23:00:51.737433 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.737348 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07038c7b-89e8-4830-949b-a0755eec2c3a-openshift-service-ca\") pod \"maas-api-e2e-api-b-774bff-f779d4d66-2tmwt\" (UID: \"07038c7b-89e8-4830-949b-a0755eec2c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt" Oct 29 23:00:51.737940 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.737916 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07038c7b-89e8-4830-949b-a0755eec2c3a-openshift-service-ca\") pod \"maas-api-e2e-api-b-774bff-f779d4d66-2tmwt\" (UID: \"07038c7b-89e8-4830-949b-a0755eec2c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt" Oct 29 23:00:51.739920 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.739896 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07038c7b-89e8-4830-949b-a0755eec2c3a-maas-api-tls\") pod \"maas-api-e2e-api-b-774bff-f779d4d66-2tmwt\" (UID: \"07038c7b-89e8-4830-949b-a0755eec2c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt" Oct 29 23:00:51.740055 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.740035 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07038c7b-89e8-4830-949b-a0755eec2c3a-metrics-certs\") pod \"maas-api-e2e-api-b-774bff-f779d4d66-2tmwt\" (UID: \"07038c7b-89e8-4830-949b-a0755eec2c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt" Oct 29 23:00:51.759924 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.759901 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z7rq\" (UniqueName: \"kubernetes.io/projected/07038c7b-89e8-4830-949b-a0755eec2c3a-kube-api-access-7z7rq\") pod \"maas-api-e2e-api-b-774bff-f779d4d66-2tmwt\" (UID: \"07038c7b-89e8-4830-949b-a0755eec2c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt" Oct 29 23:00:51.920751 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:51.920718 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt" Oct 29 23:00:52.172406 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:52.172328 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb05c692-f4a5-49bf-a84d-ca3f03edd407" path="/var/lib/kubelet/pods/cb05c692-f4a5-49bf-a84d-ca3f03edd407/volumes" Oct 29 23:00:52.243436 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:52.243406 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt"] Oct 29 23:00:52.243569 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:00:52.243459 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07038c7b_89e8_4830_949b_a0755eec2c3a.slice/crio-15d5e62e04eb03c5d2c4f4336d0aed316c41c866f7aaf73faa1d46a44b0ae613 WatchSource:0}: Error finding container 15d5e62e04eb03c5d2c4f4336d0aed316c41c866f7aaf73faa1d46a44b0ae613: Status 404 returned error can't find the container with id 15d5e62e04eb03c5d2c4f4336d0aed316c41c866f7aaf73faa1d46a44b0ae613 Oct 29 23:00:52.425502 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:52.425425 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t"] Oct 29 23:00:52.463755 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:52.463714 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs"] Oct 29 23:00:52.669771 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:52.669722 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt" event={"ID":"07038c7b-89e8-4830-949b-a0755eec2c3a","Type":"ContainerStarted","Data":"15d5e62e04eb03c5d2c4f4336d0aed316c41c866f7aaf73faa1d46a44b0ae613"} Oct 29 23:00:53.682143 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:53.682099 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt" event={"ID":"07038c7b-89e8-4830-949b-a0755eec2c3a","Type":"ContainerStarted","Data":"b593a5ae390b1af7a95d036f9bea3b62e84b4bf61ed11502be2dd946947f0af7"} Oct 29 23:00:53.682895 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:53.682185 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt" Oct 29 23:00:53.724878 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:53.724526 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt" podStartSLOduration=2.313279265 podStartE2EDuration="2.724509711s" totalImagesPullingTime="411.230446ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:53.722956473 +0000 UTC m=+1107.663504384" watchObservedRunningTime="2026-10-29 23:00:53.724509711 +0000 UTC m=+1107.665057622" Oct 29 23:00:53.895714 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:00:53.894540 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:00:57.134962 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.134924 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54"] Oct 29 23:00:57.139534 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.139511 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.156108 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.156090 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-13aa41-openshift-default-dockercfg-2zxdv\"" Oct 29 23:00:57.173342 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.173318 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54"] Oct 29 23:00:57.249519 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.249479 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-data\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.249683 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.249654 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmdl6\" (UniqueName: \"kubernetes.io/projected/14a9da2c-a970-4b40-a178-6f2215f8a44b-kube-api-access-bmdl6\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.249917 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.249891 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a9da2c-a970-4b40-a178-6f2215f8a44b-istiod-ca-cert\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.250157 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.250138 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-podinfo\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.250711 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.250282 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-envoy\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.250711 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.250339 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-token\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.250711 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.250365 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-credential-socket\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.250711 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.250401 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-workload-certs\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.250711 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.250427 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-workload-socket\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.351533 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.351489 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmdl6\" (UniqueName: \"kubernetes.io/projected/14a9da2c-a970-4b40-a178-6f2215f8a44b-kube-api-access-bmdl6\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.351740 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.351631 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a9da2c-a970-4b40-a178-6f2215f8a44b-istiod-ca-cert\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.351740 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.351672 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-podinfo\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.351861 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.351756 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-envoy\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.351861 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.351797 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-token\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.351861 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.351824 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-credential-socket\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.352014 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.351861 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-workload-certs\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.352014 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.351897 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-workload-socket\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.352119 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.352062 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-data\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.352341 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.352314 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-credential-socket\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.352452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.352360 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-workload-certs\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.352452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.352404 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-workload-socket\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.352566 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.352499 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a9da2c-a970-4b40-a178-6f2215f8a44b-istiod-ca-cert\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.352702 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.352519 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-data\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.354155 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.354136 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-envoy\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.354491 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.354464 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-podinfo\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.374237 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.374194 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-token\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.374396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.374378 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmdl6\" (UniqueName: \"kubernetes.io/projected/14a9da2c-a970-4b40-a178-6f2215f8a44b-kube-api-access-bmdl6\") pod \"e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.426248 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.426135 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t" podUID="d7012b82-2cf3-4b96-8646-2434d6aab454" containerName="payload-processing" containerID="cri-o://3fefbcc751485a109a14cf2b908053938ad195e2077b87b94020e5b15ea58763" gracePeriod=25 Oct 29 23:00:57.455115 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.455092 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:57.464547 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.464508 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs" podUID="24b8973e-4340-4632-8992-c7a80d40af90" containerName="payload-pre-processing" containerID="cri-o://a420252aafc13e35e23bdaa5fbb7369eab544dbecba138d52fced8929d03bd5e" gracePeriod=25 Oct 29 23:00:57.724683 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.724594 4821 generic.go:356] "Generic (PLEG): container finished" podID="24b8973e-4340-4632-8992-c7a80d40af90" containerID="a420252aafc13e35e23bdaa5fbb7369eab544dbecba138d52fced8929d03bd5e" exitCode=0 Oct 29 23:00:57.724899 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.724876 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs" event={"ID":"24b8973e-4340-4632-8992-c7a80d40af90","Type":"ContainerDied","Data":"a420252aafc13e35e23bdaa5fbb7369eab544dbecba138d52fced8929d03bd5e"} Oct 29 23:00:57.727004 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.726972 4821 generic.go:356] "Generic (PLEG): container finished" podID="d7012b82-2cf3-4b96-8646-2434d6aab454" containerID="3fefbcc751485a109a14cf2b908053938ad195e2077b87b94020e5b15ea58763" exitCode=0 Oct 29 23:00:57.727118 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.727040 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t" event={"ID":"d7012b82-2cf3-4b96-8646-2434d6aab454","Type":"ContainerDied","Data":"3fefbcc751485a109a14cf2b908053938ad195e2077b87b94020e5b15ea58763"} Oct 29 23:00:57.759980 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.759952 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t" Oct 29 23:00:57.788656 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.788628 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54"] Oct 29 23:00:57.789028 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:00:57.788995 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a9da2c_a970_4b40_a178_6f2215f8a44b.slice/crio-2565bdc0afd272e341feddd9b2ccc2fc521f42319e4a4409eddbf2ea410de34c WatchSource:0}: Error finding container 2565bdc0afd272e341feddd9b2ccc2fc521f42319e4a4409eddbf2ea410de34c: Status 404 returned error can't find the container with id 2565bdc0afd272e341feddd9b2ccc2fc521f42319e4a4409eddbf2ea410de34c Oct 29 23:00:57.792625 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.792590 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:00:57.792718 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.792657 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:00:57.792718 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.792688 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:00:57.853609 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.853582 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs" Oct 29 23:00:57.860621 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.860589 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7012b82-2cf3-4b96-8646-2434d6aab454-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7012b82-2cf3-4b96-8646-2434d6aab454-plugins-config-volume\") pod \"d7012b82-2cf3-4b96-8646-2434d6aab454\" (UID: \"d7012b82-2cf3-4b96-8646-2434d6aab454\") " Oct 29 23:00:57.861009 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.860981 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7012b82-2cf3-4b96-8646-2434d6aab454-kube-api-access-8p5l6\" (UniqueName: \"kubernetes.io/projected/d7012b82-2cf3-4b96-8646-2434d6aab454-kube-api-access-8p5l6\") pod \"d7012b82-2cf3-4b96-8646-2434d6aab454\" (UID: \"d7012b82-2cf3-4b96-8646-2434d6aab454\") " Oct 29 23:00:57.861109 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.861067 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7012b82-2cf3-4b96-8646-2434d6aab454-plugins-config-volume" pod "d7012b82-2cf3-4b96-8646-2434d6aab454" (UID: "d7012b82-2cf3-4b96-8646-2434d6aab454"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:57.861856 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.861830 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7012b82-2cf3-4b96-8646-2434d6aab454-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:57.868487 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.868458 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7012b82-2cf3-4b96-8646-2434d6aab454-kube-api-access-8p5l6" pod "d7012b82-2cf3-4b96-8646-2434d6aab454" (UID: "d7012b82-2cf3-4b96-8646-2434d6aab454"). InnerVolumeSpecName "kube-api-access-8p5l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:57.964392 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.963455 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b8973e-4340-4632-8992-c7a80d40af90-kube-api-access-nrl47\" (UniqueName: \"kubernetes.io/projected/24b8973e-4340-4632-8992-c7a80d40af90-kube-api-access-nrl47\") pod \"24b8973e-4340-4632-8992-c7a80d40af90\" (UID: \"24b8973e-4340-4632-8992-c7a80d40af90\") " Oct 29 23:00:57.964392 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.963727 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b8973e-4340-4632-8992-c7a80d40af90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b8973e-4340-4632-8992-c7a80d40af90-plugins-config-volume\") pod \"24b8973e-4340-4632-8992-c7a80d40af90\" (UID: \"24b8973e-4340-4632-8992-c7a80d40af90\") " Oct 29 23:00:57.964392 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.964174 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p5l6\" (UniqueName: \"kubernetes.io/projected/d7012b82-2cf3-4b96-8646-2434d6aab454-kube-api-access-8p5l6\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:57.965114 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.965032 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b8973e-4340-4632-8992-c7a80d40af90-plugins-config-volume" pod "24b8973e-4340-4632-8992-c7a80d40af90" (UID: "24b8973e-4340-4632-8992-c7a80d40af90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:57.967015 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:57.966991 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b8973e-4340-4632-8992-c7a80d40af90-kube-api-access-nrl47" pod "24b8973e-4340-4632-8992-c7a80d40af90" (UID: "24b8973e-4340-4632-8992-c7a80d40af90"). InnerVolumeSpecName "kube-api-access-nrl47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:58.065776 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:58.065688 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b8973e-4340-4632-8992-c7a80d40af90-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:58.065776 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:58.065730 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrl47\" (UniqueName: \"kubernetes.io/projected/24b8973e-4340-4632-8992-c7a80d40af90-kube-api-access-nrl47\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:00:58.742527 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:58.741472 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t" Oct 29 23:00:58.742527 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:58.742274 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t" event={"ID":"d7012b82-2cf3-4b96-8646-2434d6aab454","Type":"ContainerDied","Data":"e61936f650716ebd25cb47296dc08144ffce53b564f1853a3505454dbea3e0df"} Oct 29 23:00:58.742527 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:58.742336 4821 scope.go:122] "RemoveContainer" containerID="3fefbcc751485a109a14cf2b908053938ad195e2077b87b94020e5b15ea58763" Oct 29 23:00:58.759594 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:58.753220 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" event={"ID":"14a9da2c-a970-4b40-a178-6f2215f8a44b","Type":"ContainerStarted","Data":"5109adf583dc9f6f52b28b238df4eee33f168cee2c52d6ae3779767388b37162"} Oct 29 23:00:58.759594 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:58.753284 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" event={"ID":"14a9da2c-a970-4b40-a178-6f2215f8a44b","Type":"ContainerStarted","Data":"2565bdc0afd272e341feddd9b2ccc2fc521f42319e4a4409eddbf2ea410de34c"} Oct 29 23:00:58.759594 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:58.758652 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs" event={"ID":"24b8973e-4340-4632-8992-c7a80d40af90","Type":"ContainerDied","Data":"5f2e8ee58753c0a3fa282310499fa7c67b5105561ee7d38dd63e245adafe139b"} Oct 29 23:00:58.759594 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:58.758759 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs" Oct 29 23:00:58.768089 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:58.768065 4821 scope.go:122] "RemoveContainer" containerID="a420252aafc13e35e23bdaa5fbb7369eab544dbecba138d52fced8929d03bd5e" Oct 29 23:00:58.806516 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:58.806454 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" podStartSLOduration=1.806438078 podStartE2EDuration="1.806438078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:58.804654996 +0000 UTC m=+1112.745202907" watchObservedRunningTime="2026-10-29 23:00:58.806438078 +0000 UTC m=+1112.746986062" Oct 29 23:00:58.856356 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:58.855936 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t"] Oct 29 23:00:58.870647 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:58.865570 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-13aa41-696df59cd8-ch74t"] Oct 29 23:00:58.900828 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:58.900794 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs"] Oct 29 23:00:58.906492 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:58.906467 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-13aa41-544b9965c9-8jdxs"] Oct 29 23:00:59.455748 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:59.455711 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:00:59.457420 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:59.457393 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 29 23:00:59.457552 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:59.457440 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" podUID="14a9da2c-a970-4b40-a178-6f2215f8a44b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 29 23:00:59.695977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:00:59.695945 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt" Oct 29 23:01:00.171804 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:00.171775 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24b8973e-4340-4632-8992-c7a80d40af90" path="/var/lib/kubelet/pods/24b8973e-4340-4632-8992-c7a80d40af90/volumes" Oct 29 23:01:00.172374 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:00.172358 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7012b82-2cf3-4b96-8646-2434d6aab454" path="/var/lib/kubelet/pods/d7012b82-2cf3-4b96-8646-2434d6aab454/volumes" Oct 29 23:01:00.456104 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:00.456014 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 29 23:01:00.456104 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:00.456070 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" podUID="14a9da2c-a970-4b40-a178-6f2215f8a44b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 29 23:01:01.467050 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:01.467011 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 29 23:01:01.469988 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:01.469954 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" podUID="14a9da2c-a970-4b40-a178-6f2215f8a44b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 29 23:01:02.455874 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:02.455837 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 29 23:01:02.456067 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:02.455904 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" podUID="14a9da2c-a970-4b40-a178-6f2215f8a44b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 29 23:01:03.468225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:03.465011 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 29 23:01:03.468225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:03.465074 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" podUID="14a9da2c-a970-4b40-a178-6f2215f8a44b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 29 23:01:04.465363 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:04.464825 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 29 23:01:04.465363 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:04.464903 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" podUID="14a9da2c-a970-4b40-a178-6f2215f8a44b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 29 23:01:05.461168 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:05.461136 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:01:05.461682 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:05.461661 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:01:05.462309 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:05.462289 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:01:13.959292 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:13.959259 4821 generic.go:356] "Generic (PLEG): container finished" podID="5bfcb45f-0e94-4e58-bc91-9157be0569ed" containerID="c7c4d2d9ebfd51c6aeb68c9d6aebaaccffae8ac0057e67f724ef1bb81fd13a00" exitCode=0 Oct 29 23:01:13.959292 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:13.959297 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" event={"ID":"5bfcb45f-0e94-4e58-bc91-9157be0569ed","Type":"ContainerDied","Data":"c7c4d2d9ebfd51c6aeb68c9d6aebaaccffae8ac0057e67f724ef1bb81fd13a00"} Oct 29 23:01:13.959799 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:13.959736 4821 scope.go:122] "RemoveContainer" containerID="c7c4d2d9ebfd51c6aeb68c9d6aebaaccffae8ac0057e67f724ef1bb81fd13a00" Oct 29 23:01:13.961686 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:13.961651 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:01:13.961783 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:13.961726 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:01:13.961783 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:13.961758 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:01:14.976865 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:14.976826 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" event={"ID":"5bfcb45f-0e94-4e58-bc91-9157be0569ed","Type":"ContainerStarted","Data":"414aca8c8c390f2ed7e4acea6e90f1ed4aecd0f4fc1a0b7b18fd5d1e87ff715e"} Oct 29 23:01:15.850228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:15.843863 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:01:15.850228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:15.846017 4821 patch_prober.go:29] interesting pod/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 29 23:01:15.850228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:15.846082 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" podUID="5bfcb45f-0e94-4e58-bc91-9157be0569ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 29 23:01:16.846226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:16.845694 4821 patch_prober.go:29] interesting pod/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 29 23:01:16.846226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:16.845756 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" podUID="5bfcb45f-0e94-4e58-bc91-9157be0569ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 29 23:01:17.841746 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:17.841710 4821 patch_prober.go:29] interesting pod/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 29 23:01:17.841942 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:17.841777 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" podUID="5bfcb45f-0e94-4e58-bc91-9157be0569ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 29 23:01:18.841184 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:18.841148 4821 patch_prober.go:29] interesting pod/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 29 23:01:18.841610 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:18.841228 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" podUID="5bfcb45f-0e94-4e58-bc91-9157be0569ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 29 23:01:19.675152 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:19.670097 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv"] Oct 29 23:01:19.675152 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:19.670435 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" podUID="74ab0e3b-a727-4b4f-82b4-56712d0f448a" containerName="istio-proxy" containerID="cri-o://04fad4b405b2c3fd5dea1b9cd65bf39990b36cc9bb018d347c8ef7f72b2bb733" gracePeriod=30 Oct 29 23:01:19.846856 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:19.846340 4821 patch_prober.go:29] interesting pod/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 29 23:01:19.846856 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:19.846414 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" podUID="5bfcb45f-0e94-4e58-bc91-9157be0569ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 29 23:01:20.841785 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:20.841743 4821 patch_prober.go:29] interesting pod/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 29 23:01:20.841977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:20.841799 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" podUID="5bfcb45f-0e94-4e58-bc91-9157be0569ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 29 23:01:21.863644 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:21.863610 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:01:21.865356 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:21.865269 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:01:21.866850 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:21.866829 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:01:23.564808 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:23.564773 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv"] Oct 29 23:01:23.565349 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:23.565057 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" podUID="34564236-1496-484d-9fc9-8c35675b3dca" containerName="istio-proxy" containerID="cri-o://1c3afb1f49ba46d69558faca071166b4d79b0d2bf7db86741f435697236920b1" gracePeriod=30 Oct 29 23:01:23.771518 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:23.771466 4821 patch_prober.go:29] interesting pod/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:01:23.771685 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:23.771547 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" podUID="74ab0e3b-a727-4b4f-82b4-56712d0f448a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:01:23.908147 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:01:23.908065 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:01:24.999904 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:24.999871 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 23:01:25.099477 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.099440 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-token\" (UniqueName: \"kubernetes.io/projected/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-token\") pod \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " Oct 29 23:01:25.099688 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.099502 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-workload-socket\") pod \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " Oct 29 23:01:25.099688 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.099597 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-workload-certs\") pod \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " Oct 29 23:01:25.099688 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.099643 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istiod-ca-cert\") pod \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " Oct 29 23:01:25.099851 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.099730 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ab0e3b-a727-4b4f-82b4-56712d0f448a-kube-api-access-v8nln\" (UniqueName: \"kubernetes.io/projected/74ab0e3b-a727-4b4f-82b4-56712d0f448a-kube-api-access-v8nln\") pod \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " Oct 29 23:01:25.099851 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.099798 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-envoy\") pod \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " Oct 29 23:01:25.099950 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.099894 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-workload-socket" pod "74ab0e3b-a727-4b4f-82b4-56712d0f448a" (UID: "74ab0e3b-a727-4b4f-82b4-56712d0f448a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:25.099950 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.099938 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-workload-certs" pod "74ab0e3b-a727-4b4f-82b4-56712d0f448a" (UID: "74ab0e3b-a727-4b4f-82b4-56712d0f448a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:25.100058 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.099972 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-podinfo\") pod \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " Oct 29 23:01:25.100154 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.100111 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-data\") pod \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " Oct 29 23:01:25.100292 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.100268 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-credential-socket\") pod \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\" (UID: \"74ab0e3b-a727-4b4f-82b4-56712d0f448a\") " Oct 29 23:01:25.100600 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.100271 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istiod-ca-cert" pod "74ab0e3b-a727-4b4f-82b4-56712d0f448a" (UID: "74ab0e3b-a727-4b4f-82b4-56712d0f448a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:25.100600 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.100485 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-credential-socket" pod "74ab0e3b-a727-4b4f-82b4-56712d0f448a" (UID: "74ab0e3b-a727-4b4f-82b4-56712d0f448a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:25.100600 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.100547 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-data" pod "74ab0e3b-a727-4b4f-82b4-56712d0f448a" (UID: "74ab0e3b-a727-4b4f-82b4-56712d0f448a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:25.101398 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.101370 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:01:25.101500 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.101404 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:01:25.101500 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.101418 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:01:25.101500 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.101434 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:01:25.101500 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.101448 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:01:25.102538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.102506 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-token" pod "74ab0e3b-a727-4b4f-82b4-56712d0f448a" (UID: "74ab0e3b-a727-4b4f-82b4-56712d0f448a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:25.102642 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.102593 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ab0e3b-a727-4b4f-82b4-56712d0f448a-kube-api-access-v8nln" pod "74ab0e3b-a727-4b4f-82b4-56712d0f448a" (UID: "74ab0e3b-a727-4b4f-82b4-56712d0f448a"). InnerVolumeSpecName "kube-api-access-v8nln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:25.102986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.102961 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-envoy" pod "74ab0e3b-a727-4b4f-82b4-56712d0f448a" (UID: "74ab0e3b-a727-4b4f-82b4-56712d0f448a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:25.103386 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.103363 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-podinfo" pod "74ab0e3b-a727-4b4f-82b4-56712d0f448a" (UID: "74ab0e3b-a727-4b4f-82b4-56712d0f448a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:01:25.157920 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.157884 4821 generic.go:356] "Generic (PLEG): container finished" podID="74ab0e3b-a727-4b4f-82b4-56712d0f448a" containerID="04fad4b405b2c3fd5dea1b9cd65bf39990b36cc9bb018d347c8ef7f72b2bb733" exitCode=0 Oct 29 23:01:25.158100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.157939 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" Oct 29 23:01:25.158100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.157973 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" event={"ID":"74ab0e3b-a727-4b4f-82b4-56712d0f448a","Type":"ContainerDied","Data":"04fad4b405b2c3fd5dea1b9cd65bf39990b36cc9bb018d347c8ef7f72b2bb733"} Oct 29 23:01:25.158100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.158033 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv" event={"ID":"74ab0e3b-a727-4b4f-82b4-56712d0f448a","Type":"ContainerDied","Data":"c7c41e1281e0f20253291bc5e312d9f33fc552f603b6f54292e53d6e00bc7962"} Oct 29 23:01:25.158100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.158064 4821 scope.go:122] "RemoveContainer" containerID="04fad4b405b2c3fd5dea1b9cd65bf39990b36cc9bb018d347c8ef7f72b2bb733" Oct 29 23:01:25.169975 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.169954 4821 scope.go:122] "RemoveContainer" containerID="04fad4b405b2c3fd5dea1b9cd65bf39990b36cc9bb018d347c8ef7f72b2bb733" Oct 29 23:01:25.170278 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:01:25.170252 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04fad4b405b2c3fd5dea1b9cd65bf39990b36cc9bb018d347c8ef7f72b2bb733\": container with ID starting with 04fad4b405b2c3fd5dea1b9cd65bf39990b36cc9bb018d347c8ef7f72b2bb733 not found: ID does not exist" containerID="04fad4b405b2c3fd5dea1b9cd65bf39990b36cc9bb018d347c8ef7f72b2bb733" Oct 29 23:01:25.170394 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.170284 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04fad4b405b2c3fd5dea1b9cd65bf39990b36cc9bb018d347c8ef7f72b2bb733"} err="failed to get container status \"04fad4b405b2c3fd5dea1b9cd65bf39990b36cc9bb018d347c8ef7f72b2bb733\": rpc error: code = NotFound desc = could not find container \"04fad4b405b2c3fd5dea1b9cd65bf39990b36cc9bb018d347c8ef7f72b2bb733\": container with ID starting with 04fad4b405b2c3fd5dea1b9cd65bf39990b36cc9bb018d347c8ef7f72b2bb733 not found: ID does not exist" Oct 29 23:01:25.243584 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.239797 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8nln\" (UniqueName: \"kubernetes.io/projected/74ab0e3b-a727-4b4f-82b4-56712d0f448a-kube-api-access-v8nln\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:01:25.243584 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.239834 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:01:25.243584 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.239852 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:01:25.243584 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.239864 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ab0e3b-a727-4b4f-82b4-56712d0f448a-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:01:25.259364 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.259328 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv"] Oct 29 23:01:25.267397 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:25.267348 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f3092a4d-openshift-default-6cd65bcfd9-8bmkv"] Oct 29 23:01:26.222297 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:26.222259 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ab0e3b-a727-4b4f-82b4-56712d0f448a" path="/var/lib/kubelet/pods/74ab0e3b-a727-4b4f-82b4-56712d0f448a/volumes" Oct 29 23:01:28.898889 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:28.898789 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 23:01:28.947085 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:28.947049 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34564236-1496-484d-9fc9-8c35675b3dca-kube-api-access-rvbq4\" (UniqueName: \"kubernetes.io/projected/34564236-1496-484d-9fc9-8c35675b3dca-kube-api-access-rvbq4\") pod \"34564236-1496-484d-9fc9-8c35675b3dca\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " Oct 29 23:01:28.947288 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:28.947125 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-istio-envoy\") pod \"34564236-1496-484d-9fc9-8c35675b3dca\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " Oct 29 23:01:28.947288 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:28.947173 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34564236-1496-484d-9fc9-8c35675b3dca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34564236-1496-484d-9fc9-8c35675b3dca-istio-podinfo\") pod \"34564236-1496-484d-9fc9-8c35675b3dca\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " Oct 29 23:01:28.947288 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:28.947278 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-workload-socket\") pod \"34564236-1496-484d-9fc9-8c35675b3dca\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " Oct 29 23:01:28.947465 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:28.947359 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-credential-socket\") pod \"34564236-1496-484d-9fc9-8c35675b3dca\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " Oct 29 23:01:28.947465 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:28.947401 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-workload-certs\") pod \"34564236-1496-484d-9fc9-8c35675b3dca\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " Oct 29 23:01:28.947710 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:28.947512 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34564236-1496-484d-9fc9-8c35675b3dca-istio-token\" (UniqueName: \"kubernetes.io/projected/34564236-1496-484d-9fc9-8c35675b3dca-istio-token\") pod \"34564236-1496-484d-9fc9-8c35675b3dca\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " Oct 29 23:01:28.947710 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:28.947593 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-istio-data\") pod \"34564236-1496-484d-9fc9-8c35675b3dca\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " Oct 29 23:01:28.947710 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:28.947641 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34564236-1496-484d-9fc9-8c35675b3dca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34564236-1496-484d-9fc9-8c35675b3dca-istiod-ca-cert\") pod \"34564236-1496-484d-9fc9-8c35675b3dca\" (UID: \"34564236-1496-484d-9fc9-8c35675b3dca\") " Oct 29 23:01:28.947892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:28.947592 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-workload-socket" pod "34564236-1496-484d-9fc9-8c35675b3dca" (UID: "34564236-1496-484d-9fc9-8c35675b3dca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:28.947892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:28.947641 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-credential-socket" pod "34564236-1496-484d-9fc9-8c35675b3dca" (UID: "34564236-1496-484d-9fc9-8c35675b3dca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:28.947892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:28.947722 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-workload-certs" pod "34564236-1496-484d-9fc9-8c35675b3dca" (UID: "34564236-1496-484d-9fc9-8c35675b3dca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:28.947892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:28.947842 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-istio-data" pod "34564236-1496-484d-9fc9-8c35675b3dca" (UID: "34564236-1496-484d-9fc9-8c35675b3dca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:28.948380 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:28.948240 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34564236-1496-484d-9fc9-8c35675b3dca-istiod-ca-cert" pod "34564236-1496-484d-9fc9-8c35675b3dca" (UID: "34564236-1496-484d-9fc9-8c35675b3dca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:28.948554 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:28.948530 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.948554 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:28.948555 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.948554 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:28.948570 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.948761 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:28.948584 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.948761 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:28.948596 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34564236-1496-484d-9fc9-8c35675b3dca-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.956227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:28.952558 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34564236-1496-484d-9fc9-8c35675b3dca-kube-api-access-rvbq4" pod "34564236-1496-484d-9fc9-8c35675b3dca" (UID: "34564236-1496-484d-9fc9-8c35675b3dca"). InnerVolumeSpecName "kube-api-access-rvbq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:28.956227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:28.952601 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34564236-1496-484d-9fc9-8c35675b3dca-istio-podinfo" pod "34564236-1496-484d-9fc9-8c35675b3dca" (UID: "34564236-1496-484d-9fc9-8c35675b3dca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:01:28.956227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:28.952689 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-istio-envoy" pod "34564236-1496-484d-9fc9-8c35675b3dca" (UID: "34564236-1496-484d-9fc9-8c35675b3dca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:28.956227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:28.953286 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34564236-1496-484d-9fc9-8c35675b3dca-istio-token" pod "34564236-1496-484d-9fc9-8c35675b3dca" (UID: "34564236-1496-484d-9fc9-8c35675b3dca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:29.049809 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:29.049777 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34564236-1496-484d-9fc9-8c35675b3dca-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:01:29.049809 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:29.049806 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvbq4\" (UniqueName: \"kubernetes.io/projected/34564236-1496-484d-9fc9-8c35675b3dca-kube-api-access-rvbq4\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:01:29.049809 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:29.049816 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34564236-1496-484d-9fc9-8c35675b3dca-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:01:29.050072 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:29.049826 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34564236-1496-484d-9fc9-8c35675b3dca-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:01:29.257298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:29.257267 4821 generic.go:356] "Generic (PLEG): container finished" podID="34564236-1496-484d-9fc9-8c35675b3dca" containerID="1c3afb1f49ba46d69558faca071166b4d79b0d2bf7db86741f435697236920b1" exitCode=0 Oct 29 23:01:29.257481 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:29.257321 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" Oct 29 23:01:29.257481 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:29.257329 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" event={"ID":"34564236-1496-484d-9fc9-8c35675b3dca","Type":"ContainerDied","Data":"1c3afb1f49ba46d69558faca071166b4d79b0d2bf7db86741f435697236920b1"} Oct 29 23:01:29.257481 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:29.257375 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv" event={"ID":"34564236-1496-484d-9fc9-8c35675b3dca","Type":"ContainerDied","Data":"5b0f1c981ab5ea51c17bd69e10cac9573f0325c922585742b04fb22a13ebcac6"} Oct 29 23:01:29.257481 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:29.257395 4821 scope.go:122] "RemoveContainer" containerID="1c3afb1f49ba46d69558faca071166b4d79b0d2bf7db86741f435697236920b1" Oct 29 23:01:29.269679 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:29.269660 4821 scope.go:122] "RemoveContainer" containerID="d01cd83e27d2729b28c647d5632daf26ad1bbbba5cc498bcb284a9e97b9f3c66" Oct 29 23:01:29.281229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:29.280753 4821 scope.go:122] "RemoveContainer" containerID="1c3afb1f49ba46d69558faca071166b4d79b0d2bf7db86741f435697236920b1" Oct 29 23:01:29.283872 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:01:29.282384 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c3afb1f49ba46d69558faca071166b4d79b0d2bf7db86741f435697236920b1\": container with ID starting with 1c3afb1f49ba46d69558faca071166b4d79b0d2bf7db86741f435697236920b1 not found: ID does not exist" containerID="1c3afb1f49ba46d69558faca071166b4d79b0d2bf7db86741f435697236920b1" Oct 29 23:01:29.283872 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:29.282425 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3afb1f49ba46d69558faca071166b4d79b0d2bf7db86741f435697236920b1"} err="failed to get container status \"1c3afb1f49ba46d69558faca071166b4d79b0d2bf7db86741f435697236920b1\": rpc error: code = NotFound desc = could not find container \"1c3afb1f49ba46d69558faca071166b4d79b0d2bf7db86741f435697236920b1\": container with ID starting with 1c3afb1f49ba46d69558faca071166b4d79b0d2bf7db86741f435697236920b1 not found: ID does not exist" Oct 29 23:01:29.283872 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:29.282449 4821 scope.go:122] "RemoveContainer" containerID="d01cd83e27d2729b28c647d5632daf26ad1bbbba5cc498bcb284a9e97b9f3c66" Oct 29 23:01:29.283872 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:01:29.283484 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d01cd83e27d2729b28c647d5632daf26ad1bbbba5cc498bcb284a9e97b9f3c66\": container with ID starting with d01cd83e27d2729b28c647d5632daf26ad1bbbba5cc498bcb284a9e97b9f3c66 not found: ID does not exist" containerID="d01cd83e27d2729b28c647d5632daf26ad1bbbba5cc498bcb284a9e97b9f3c66" Oct 29 23:01:29.283872 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:29.283524 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d01cd83e27d2729b28c647d5632daf26ad1bbbba5cc498bcb284a9e97b9f3c66"} err="failed to get container status \"d01cd83e27d2729b28c647d5632daf26ad1bbbba5cc498bcb284a9e97b9f3c66\": rpc error: code = NotFound desc = could not find container \"d01cd83e27d2729b28c647d5632daf26ad1bbbba5cc498bcb284a9e97b9f3c66\": container with ID starting with d01cd83e27d2729b28c647d5632daf26ad1bbbba5cc498bcb284a9e97b9f3c66 not found: ID does not exist" Oct 29 23:01:29.312682 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:29.312658 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv"] Oct 29 23:01:29.345368 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:29.345324 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0df9b0-openshift-default-7798844b69-6w6qv"] Oct 29 23:01:30.172264 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:30.172181 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34564236-1496-484d-9fc9-8c35675b3dca" path="/var/lib/kubelet/pods/34564236-1496-484d-9fc9-8c35675b3dca/volumes" Oct 29 23:01:31.545671 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.545631 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs"] Oct 29 23:01:31.546355 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.546234 4821 cpu_manager.go:414] "containerMap: removing container" podUID="34564236-1496-484d-9fc9-8c35675b3dca" containerName="istio-proxy" Oct 29 23:01:31.546355 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.546247 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34564236-1496-484d-9fc9-8c35675b3dca" containerName="istio-proxy" Oct 29 23:01:31.546355 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.546267 4821 cpu_manager.go:414] "containerMap: removing container" podUID="d7012b82-2cf3-4b96-8646-2434d6aab454" containerName="payload-processing" Oct 29 23:01:31.546355 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.546272 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7012b82-2cf3-4b96-8646-2434d6aab454" containerName="payload-processing" Oct 29 23:01:31.546355 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.546290 4821 cpu_manager.go:414] "containerMap: removing container" podUID="34564236-1496-484d-9fc9-8c35675b3dca" containerName="istio-proxy" Oct 29 23:01:31.546355 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.546294 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34564236-1496-484d-9fc9-8c35675b3dca" containerName="istio-proxy" Oct 29 23:01:31.546355 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.546314 4821 cpu_manager.go:414] "containerMap: removing container" podUID="74ab0e3b-a727-4b4f-82b4-56712d0f448a" containerName="istio-proxy" Oct 29 23:01:31.546355 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.546320 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ab0e3b-a727-4b4f-82b4-56712d0f448a" containerName="istio-proxy" Oct 29 23:01:31.546355 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.546332 4821 cpu_manager.go:414] "containerMap: removing container" podUID="24b8973e-4340-4632-8992-c7a80d40af90" containerName="payload-pre-processing" Oct 29 23:01:31.546355 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.546337 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b8973e-4340-4632-8992-c7a80d40af90" containerName="payload-pre-processing" Oct 29 23:01:31.547001 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.546452 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="34564236-1496-484d-9fc9-8c35675b3dca" containerName="istio-proxy" Oct 29 23:01:31.547001 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.546461 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7012b82-2cf3-4b96-8646-2434d6aab454" containerName="payload-processing" Oct 29 23:01:31.547001 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.546473 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ab0e3b-a727-4b4f-82b4-56712d0f448a" containerName="istio-proxy" Oct 29 23:01:31.547001 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.546481 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b8973e-4340-4632-8992-c7a80d40af90" containerName="payload-pre-processing" Oct 29 23:01:31.551193 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.551171 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs" Oct 29 23:01:31.581031 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.581000 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs"] Oct 29 23:01:31.585290 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.585264 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-0266dd\"" Oct 29 23:01:31.585479 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.585368 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-0266dd-dockercfg-jq4hc\"" Oct 29 23:01:31.600530 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.600493 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhwcn\" (UniqueName: \"kubernetes.io/projected/3c7c4b21-7397-4db3-b170-aaded0958eaa-kube-api-access-bhwcn\") pod \"payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs\" (UID: \"3c7c4b21-7397-4db3-b170-aaded0958eaa\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs" Oct 29 23:01:31.600782 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.600760 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c7c4b21-7397-4db3-b170-aaded0958eaa-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs\" (UID: \"3c7c4b21-7397-4db3-b170-aaded0958eaa\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs" Oct 29 23:01:31.769228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.705852 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhwcn\" (UniqueName: \"kubernetes.io/projected/3c7c4b21-7397-4db3-b170-aaded0958eaa-kube-api-access-bhwcn\") pod \"payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs\" (UID: \"3c7c4b21-7397-4db3-b170-aaded0958eaa\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs" Oct 29 23:01:31.769228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.763737 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c7c4b21-7397-4db3-b170-aaded0958eaa-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs\" (UID: \"3c7c4b21-7397-4db3-b170-aaded0958eaa\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs" Oct 29 23:01:31.788224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.782151 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c7c4b21-7397-4db3-b170-aaded0958eaa-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs\" (UID: \"3c7c4b21-7397-4db3-b170-aaded0958eaa\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs" Oct 29 23:01:31.832275 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.829375 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhwcn\" (UniqueName: \"kubernetes.io/projected/3c7c4b21-7397-4db3-b170-aaded0958eaa-kube-api-access-bhwcn\") pod \"payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs\" (UID: \"3c7c4b21-7397-4db3-b170-aaded0958eaa\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs" Oct 29 23:01:31.873595 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:31.873538 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs" Oct 29 23:01:32.348390 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:32.348358 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs"] Oct 29 23:01:32.349932 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:01:32.349903 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c7c4b21_7397_4db3_b170_aaded0958eaa.slice/crio-5a19fa9c41e7c5af70c4b2bd2065133215d220263d800195d2c1903d8dddb009 WatchSource:0}: Error finding container 5a19fa9c41e7c5af70c4b2bd2065133215d220263d800195d2c1903d8dddb009: Status 404 returned error can't find the container with id 5a19fa9c41e7c5af70c4b2bd2065133215d220263d800195d2c1903d8dddb009 Oct 29 23:01:33.294542 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:33.294501 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs" event={"ID":"3c7c4b21-7397-4db3-b170-aaded0958eaa","Type":"ContainerStarted","Data":"701a35a9df18d5b8eaebc2c0e79a51e7c183b8707fbaf3a99d1dbda478a7cc57"} Oct 29 23:01:33.294933 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:33.294705 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs" Oct 29 23:01:33.294933 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:33.294738 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs" event={"ID":"3c7c4b21-7397-4db3-b170-aaded0958eaa","Type":"ContainerStarted","Data":"5a19fa9c41e7c5af70c4b2bd2065133215d220263d800195d2c1903d8dddb009"} Oct 29 23:01:33.336990 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:33.336939 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs" podStartSLOduration=2.33692566 podStartE2EDuration="2.33692566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:33.335132288 +0000 UTC m=+1147.275680199" watchObservedRunningTime="2026-10-29 23:01:33.33692566 +0000 UTC m=+1147.277473570" Oct 29 23:01:36.825080 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:36.823289 4821 patch_prober.go:29] interesting pod/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": read tcp 10.133.0.2:53352->10.133.0.121:15021: read: connection reset by peer" start-of-body= Oct 29 23:01:36.825080 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:36.823369 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" podUID="59c2911d-7186-44a7-9b7b-e5c55f1c185a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": read tcp 10.133.0.2:53352->10.133.0.121:15021: read: connection reset by peer" Oct 29 23:01:37.327769 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:37.327739 4821 generic.go:356] "Generic (PLEG): container finished" podID="59c2911d-7186-44a7-9b7b-e5c55f1c185a" containerID="9cd89555453f6592bcd7e640bbbe6c16d378d123539af7cda8219930f3338018" exitCode=0 Oct 29 23:01:37.327944 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:37.327821 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" event={"ID":"59c2911d-7186-44a7-9b7b-e5c55f1c185a","Type":"ContainerDied","Data":"9cd89555453f6592bcd7e640bbbe6c16d378d123539af7cda8219930f3338018"} Oct 29 23:01:37.328340 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:37.328323 4821 scope.go:122] "RemoveContainer" containerID="9cd89555453f6592bcd7e640bbbe6c16d378d123539af7cda8219930f3338018" Oct 29 23:01:37.330152 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:37.330127 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:01:37.330241 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:37.330192 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:01:37.330289 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:37.330262 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:01:38.337266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:38.337228 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" event={"ID":"59c2911d-7186-44a7-9b7b-e5c55f1c185a","Type":"ContainerStarted","Data":"e06b4d41461dc76cc60571ca0c826e5e8b45a1e93126572c4085192a39ef9f5c"} Oct 29 23:01:38.445908 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:38.445874 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:01:38.446116 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:38.446095 4821 patch_prober.go:29] interesting pod/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 29 23:01:38.446241 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:38.446163 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" podUID="59c2911d-7186-44a7-9b7b-e5c55f1c185a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 29 23:01:39.446914 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:39.446862 4821 patch_prober.go:29] interesting pod/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 29 23:01:39.447459 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:39.446939 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" podUID="59c2911d-7186-44a7-9b7b-e5c55f1c185a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 29 23:01:40.447031 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:40.446994 4821 patch_prober.go:29] interesting pod/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 29 23:01:40.447519 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:40.447062 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" podUID="59c2911d-7186-44a7-9b7b-e5c55f1c185a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 29 23:01:41.446777 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:41.446745 4821 patch_prober.go:29] interesting pod/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 29 23:01:41.446961 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:41.446811 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" podUID="59c2911d-7186-44a7-9b7b-e5c55f1c185a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 29 23:01:42.446735 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:42.446702 4821 patch_prober.go:29] interesting pod/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 29 23:01:42.447143 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:42.446756 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" podUID="59c2911d-7186-44a7-9b7b-e5c55f1c185a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 29 23:01:43.446783 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:43.446742 4821 patch_prober.go:29] interesting pod/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 29 23:01:43.447298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:43.446797 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" podUID="59c2911d-7186-44a7-9b7b-e5c55f1c185a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 29 23:01:44.306573 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:44.306538 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs" Oct 29 23:01:44.451217 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:44.451173 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:01:44.451664 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:44.451520 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:01:44.452194 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:44.452170 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:01:52.038401 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:52.038368 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af65e63-nndvh"] Oct 29 23:01:52.050315 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:52.050288 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af65e63-nndvh"] Oct 29 23:01:52.170987 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:01:52.170957 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c07ef42d-e6d7-423c-8424-62efb7d851d7" path="/var/lib/kubelet/pods/c07ef42d-e6d7-423c-8424-62efb7d851d7/volumes" Oct 29 23:01:53.890513 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:01:53.890472 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:02:05.183472 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:05.183441 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r"] Oct 29 23:02:05.184174 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:05.184159 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="34564236-1496-484d-9fc9-8c35675b3dca" containerName="istio-proxy" Oct 29 23:02:05.188652 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:05.188634 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r" Oct 29 23:02:05.204159 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:05.204132 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:02:05.217472 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:05.216801 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:02:05.223250 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:05.221095 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1299a73b-9d42-44dc-aa4c-e535358ba2c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r\" (UID: \"1299a73b-9d42-44dc-aa4c-e535358ba2c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r" Oct 29 23:02:05.223250 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:05.221158 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f7fp\" (UniqueName: \"kubernetes.io/projected/1299a73b-9d42-44dc-aa4c-e535358ba2c5-kube-api-access-6f7fp\") pod \"maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r\" (UID: \"1299a73b-9d42-44dc-aa4c-e535358ba2c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r" Oct 29 23:02:05.223250 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:05.221443 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1299a73b-9d42-44dc-aa4c-e535358ba2c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r\" (UID: \"1299a73b-9d42-44dc-aa4c-e535358ba2c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r" Oct 29 23:02:05.224214 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:05.224173 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r"] Oct 29 23:02:05.322885 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:05.322847 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1299a73b-9d42-44dc-aa4c-e535358ba2c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r\" (UID: \"1299a73b-9d42-44dc-aa4c-e535358ba2c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r" Oct 29 23:02:05.323063 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:05.323006 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1299a73b-9d42-44dc-aa4c-e535358ba2c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r\" (UID: \"1299a73b-9d42-44dc-aa4c-e535358ba2c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r" Oct 29 23:02:05.323063 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:05.323036 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f7fp\" (UniqueName: \"kubernetes.io/projected/1299a73b-9d42-44dc-aa4c-e535358ba2c5-kube-api-access-6f7fp\") pod \"maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r\" (UID: \"1299a73b-9d42-44dc-aa4c-e535358ba2c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r" Oct 29 23:02:05.323876 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:05.323779 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1299a73b-9d42-44dc-aa4c-e535358ba2c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r\" (UID: \"1299a73b-9d42-44dc-aa4c-e535358ba2c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r" Oct 29 23:02:05.325647 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:05.325620 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1299a73b-9d42-44dc-aa4c-e535358ba2c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r\" (UID: \"1299a73b-9d42-44dc-aa4c-e535358ba2c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r" Oct 29 23:02:05.351449 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:05.348105 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f7fp\" (UniqueName: \"kubernetes.io/projected/1299a73b-9d42-44dc-aa4c-e535358ba2c5-kube-api-access-6f7fp\") pod \"maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r\" (UID: \"1299a73b-9d42-44dc-aa4c-e535358ba2c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r" Oct 29 23:02:05.507329 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:05.507185 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r" Oct 29 23:02:05.859116 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:05.859089 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r"] Oct 29 23:02:05.860464 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:02:05.860441 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1299a73b_9d42_44dc_aa4c_e535358ba2c5.slice/crio-0c309c791267fe1d6b446078a49fc8c242abec718713f0bf971170b1efbc84d4 WatchSource:0}: Error finding container 0c309c791267fe1d6b446078a49fc8c242abec718713f0bf971170b1efbc84d4: Status 404 returned error can't find the container with id 0c309c791267fe1d6b446078a49fc8c242abec718713f0bf971170b1efbc84d4 Oct 29 23:02:06.655012 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:06.654961 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r" event={"ID":"1299a73b-9d42-44dc-aa4c-e535358ba2c5","Type":"ContainerStarted","Data":"6dc55bf1f444b03094c2682fb7e5a033a12d23e245575017542c4be9d529a35e"} Oct 29 23:02:06.655477 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:06.655035 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r" event={"ID":"1299a73b-9d42-44dc-aa4c-e535358ba2c5","Type":"ContainerStarted","Data":"0c309c791267fe1d6b446078a49fc8c242abec718713f0bf971170b1efbc84d4"} Oct 29 23:02:06.699661 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:06.699611 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r" podStartSLOduration=1.699597722 podStartE2EDuration="1.699597722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:06.69425887 +0000 UTC m=+1180.634806783" watchObservedRunningTime="2026-10-29 23:02:06.699597722 +0000 UTC m=+1180.640145632" Oct 29 23:02:07.665933 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:07.665881 4821 generic.go:356] "Generic (PLEG): container finished" podID="1299a73b-9d42-44dc-aa4c-e535358ba2c5" containerID="6dc55bf1f444b03094c2682fb7e5a033a12d23e245575017542c4be9d529a35e" exitCode=0 Oct 29 23:02:07.666543 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:07.666270 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r" event={"ID":"1299a73b-9d42-44dc-aa4c-e535358ba2c5","Type":"ContainerDied","Data":"6dc55bf1f444b03094c2682fb7e5a033a12d23e245575017542c4be9d529a35e"} Oct 29 23:02:08.887907 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:08.887877 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r" Oct 29 23:02:08.926361 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:08.926280 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh"] Oct 29 23:02:08.926510 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:08.926474 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" podUID="0c2e7b94-f0cf-424d-a296-c2fe8e60ff01" containerName="istio-proxy" containerID="cri-o://a8c7cdf4c3a29e9f2627730816b381e54a3446275af17a1f0f7fc7f0d4ffbf31" gracePeriod=30 Oct 29 23:02:08.940233 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:08.939812 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42"] Oct 29 23:02:08.940233 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:08.940077 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" podUID="0bb6b077-1785-4673-9dc9-80258dcbecf3" containerName="istio-proxy" containerID="cri-o://c56926b65dfa34cafd80fac4c455899fe4bf67e7a20f0cc082eb47b0e159fa6d" gracePeriod=30 Oct 29 23:02:08.944745 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:08.944719 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn"] Oct 29 23:02:08.944956 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:08.944931 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" podUID="6103a8bf-2a28-4991-8b55-fecb16515d7f" containerName="istio-proxy" containerID="cri-o://23cf050765136816912dd67848761598c1b3143d1d17aa6109ba0e68ae56ec19" gracePeriod=30 Oct 29 23:02:08.975697 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:08.975659 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:02:08.975896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:08.975741 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" podUID="0bb6b077-1785-4673-9dc9-80258dcbecf3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:02:08.991947 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:08.991914 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:02:08.992111 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:08.991983 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" podUID="6103a8bf-2a28-4991-8b55-fecb16515d7f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:02:09.006008 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:09.005980 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1299a73b-9d42-44dc-aa4c-e535358ba2c5-kube-api-access-6f7fp\" (UniqueName: \"kubernetes.io/projected/1299a73b-9d42-44dc-aa4c-e535358ba2c5-kube-api-access-6f7fp\") pod \"1299a73b-9d42-44dc-aa4c-e535358ba2c5\" (UID: \"1299a73b-9d42-44dc-aa4c-e535358ba2c5\") " Oct 29 23:02:09.006284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:09.006075 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1299a73b-9d42-44dc-aa4c-e535358ba2c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1299a73b-9d42-44dc-aa4c-e535358ba2c5-maas-api-xks-ca\") pod \"1299a73b-9d42-44dc-aa4c-e535358ba2c5\" (UID: \"1299a73b-9d42-44dc-aa4c-e535358ba2c5\") " Oct 29 23:02:09.006284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:09.006169 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1299a73b-9d42-44dc-aa4c-e535358ba2c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1299a73b-9d42-44dc-aa4c-e535358ba2c5-maas-api-openshift-ca\") pod \"1299a73b-9d42-44dc-aa4c-e535358ba2c5\" (UID: \"1299a73b-9d42-44dc-aa4c-e535358ba2c5\") " Oct 29 23:02:09.006986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:09.006955 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1299a73b-9d42-44dc-aa4c-e535358ba2c5-maas-api-openshift-ca" pod "1299a73b-9d42-44dc-aa4c-e535358ba2c5" (UID: "1299a73b-9d42-44dc-aa4c-e535358ba2c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:09.007260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:09.007237 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1299a73b-9d42-44dc-aa4c-e535358ba2c5-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:09.009098 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:09.009075 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1299a73b-9d42-44dc-aa4c-e535358ba2c5-maas-api-xks-ca" pod "1299a73b-9d42-44dc-aa4c-e535358ba2c5" (UID: "1299a73b-9d42-44dc-aa4c-e535358ba2c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:02:09.009434 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:09.009408 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1299a73b-9d42-44dc-aa4c-e535358ba2c5-kube-api-access-6f7fp" pod "1299a73b-9d42-44dc-aa4c-e535358ba2c5" (UID: "1299a73b-9d42-44dc-aa4c-e535358ba2c5"). InnerVolumeSpecName "kube-api-access-6f7fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:09.108719 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:09.108681 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f7fp\" (UniqueName: \"kubernetes.io/projected/1299a73b-9d42-44dc-aa4c-e535358ba2c5-kube-api-access-6f7fp\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:09.108719 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:09.108717 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1299a73b-9d42-44dc-aa4c-e535358ba2c5-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:09.682913 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:09.682885 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r" Oct 29 23:02:09.682913 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:09.682905 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r" event={"ID":"1299a73b-9d42-44dc-aa4c-e535358ba2c5","Type":"ContainerDied","Data":"0c309c791267fe1d6b446078a49fc8c242abec718713f0bf971170b1efbc84d4"} Oct 29 23:02:09.683190 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:09.682960 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c309c791267fe1d6b446078a49fc8c242abec718713f0bf971170b1efbc84d4" Oct 29 23:02:10.720563 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:10.720532 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w"] Oct 29 23:02:10.721398 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:10.721374 4821 cpu_manager.go:414] "containerMap: removing container" podUID="1299a73b-9d42-44dc-aa4c-e535358ba2c5" containerName="revoke-keys" Oct 29 23:02:10.721398 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:10.721396 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1299a73b-9d42-44dc-aa4c-e535358ba2c5" containerName="revoke-keys" Oct 29 23:02:10.721587 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:10.721538 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="1299a73b-9d42-44dc-aa4c-e535358ba2c5" containerName="revoke-keys" Oct 29 23:02:10.725038 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:10.725015 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w" Oct 29 23:02:10.746504 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:10.746477 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w"] Oct 29 23:02:10.753871 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:10.753847 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:02:10.754015 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:10.753910 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:02:10.853667 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:10.853629 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ls7c\" (UniqueName: \"kubernetes.io/projected/91606ba2-0df7-4092-8166-2e6c65bbfea2-kube-api-access-4ls7c\") pod \"maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w\" (UID: \"91606ba2-0df7-4092-8166-2e6c65bbfea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w" Oct 29 23:02:10.853815 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:10.853793 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91606ba2-0df7-4092-8166-2e6c65bbfea2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w\" (UID: \"91606ba2-0df7-4092-8166-2e6c65bbfea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w" Oct 29 23:02:10.853984 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:10.853968 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91606ba2-0df7-4092-8166-2e6c65bbfea2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w\" (UID: \"91606ba2-0df7-4092-8166-2e6c65bbfea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w" Oct 29 23:02:10.955381 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:10.955346 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ls7c\" (UniqueName: \"kubernetes.io/projected/91606ba2-0df7-4092-8166-2e6c65bbfea2-kube-api-access-4ls7c\") pod \"maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w\" (UID: \"91606ba2-0df7-4092-8166-2e6c65bbfea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w" Oct 29 23:02:10.955586 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:10.955497 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91606ba2-0df7-4092-8166-2e6c65bbfea2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w\" (UID: \"91606ba2-0df7-4092-8166-2e6c65bbfea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w" Oct 29 23:02:10.955714 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:10.955695 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91606ba2-0df7-4092-8166-2e6c65bbfea2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w\" (UID: \"91606ba2-0df7-4092-8166-2e6c65bbfea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w" Oct 29 23:02:10.956149 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:10.956130 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91606ba2-0df7-4092-8166-2e6c65bbfea2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w\" (UID: \"91606ba2-0df7-4092-8166-2e6c65bbfea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w" Oct 29 23:02:10.958177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:10.958156 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91606ba2-0df7-4092-8166-2e6c65bbfea2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w\" (UID: \"91606ba2-0df7-4092-8166-2e6c65bbfea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w" Oct 29 23:02:10.980041 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:10.979951 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ls7c\" (UniqueName: \"kubernetes.io/projected/91606ba2-0df7-4092-8166-2e6c65bbfea2-kube-api-access-4ls7c\") pod \"maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w\" (UID: \"91606ba2-0df7-4092-8166-2e6c65bbfea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w" Oct 29 23:02:11.039518 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:11.039459 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w" Oct 29 23:02:11.381274 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:11.381235 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w"] Oct 29 23:02:11.388229 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:02:11.385953 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91606ba2_0df7_4092_8166_2e6c65bbfea2.slice/crio-95353ac18faa6e8392b01d261e0c07abdcf1919f90170f79562fe251d9b63e66 WatchSource:0}: Error finding container 95353ac18faa6e8392b01d261e0c07abdcf1919f90170f79562fe251d9b63e66: Status 404 returned error can't find the container with id 95353ac18faa6e8392b01d261e0c07abdcf1919f90170f79562fe251d9b63e66 Oct 29 23:02:11.700957 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:11.700925 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w" event={"ID":"91606ba2-0df7-4092-8166-2e6c65bbfea2","Type":"ContainerStarted","Data":"95353ac18faa6e8392b01d261e0c07abdcf1919f90170f79562fe251d9b63e66"} Oct 29 23:02:12.710501 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:12.710412 4821 generic.go:356] "Generic (PLEG): container finished" podID="91606ba2-0df7-4092-8166-2e6c65bbfea2" containerID="8f9a4ad0a295170e51f44372908d585faa6efa233e0998901b44703fa6fb63d1" exitCode=0 Oct 29 23:02:12.710866 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:12.710498 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w" event={"ID":"91606ba2-0df7-4092-8166-2e6c65bbfea2","Type":"ContainerDied","Data":"8f9a4ad0a295170e51f44372908d585faa6efa233e0998901b44703fa6fb63d1"} Oct 29 23:02:13.884774 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:13.884744 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w" Oct 29 23:02:14.009919 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.009886 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91606ba2-0df7-4092-8166-2e6c65bbfea2-kube-api-access-4ls7c\" (UniqueName: \"kubernetes.io/projected/91606ba2-0df7-4092-8166-2e6c65bbfea2-kube-api-access-4ls7c\") pod \"91606ba2-0df7-4092-8166-2e6c65bbfea2\" (UID: \"91606ba2-0df7-4092-8166-2e6c65bbfea2\") " Oct 29 23:02:14.010059 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.010019 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91606ba2-0df7-4092-8166-2e6c65bbfea2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91606ba2-0df7-4092-8166-2e6c65bbfea2-maas-api-xks-ca\") pod \"91606ba2-0df7-4092-8166-2e6c65bbfea2\" (UID: \"91606ba2-0df7-4092-8166-2e6c65bbfea2\") " Oct 29 23:02:14.010269 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.010252 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91606ba2-0df7-4092-8166-2e6c65bbfea2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91606ba2-0df7-4092-8166-2e6c65bbfea2-maas-api-openshift-ca\") pod \"91606ba2-0df7-4092-8166-2e6c65bbfea2\" (UID: \"91606ba2-0df7-4092-8166-2e6c65bbfea2\") " Oct 29 23:02:14.010613 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.010590 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91606ba2-0df7-4092-8166-2e6c65bbfea2-maas-api-openshift-ca" pod "91606ba2-0df7-4092-8166-2e6c65bbfea2" (UID: "91606ba2-0df7-4092-8166-2e6c65bbfea2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:14.010914 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.010893 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91606ba2-0df7-4092-8166-2e6c65bbfea2-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.012756 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.012697 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91606ba2-0df7-4092-8166-2e6c65bbfea2-maas-api-xks-ca" pod "91606ba2-0df7-4092-8166-2e6c65bbfea2" (UID: "91606ba2-0df7-4092-8166-2e6c65bbfea2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:02:14.012756 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.012717 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91606ba2-0df7-4092-8166-2e6c65bbfea2-kube-api-access-4ls7c" pod "91606ba2-0df7-4092-8166-2e6c65bbfea2" (UID: "91606ba2-0df7-4092-8166-2e6c65bbfea2"). InnerVolumeSpecName "kube-api-access-4ls7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:14.113080 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.113045 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ls7c\" (UniqueName: \"kubernetes.io/projected/91606ba2-0df7-4092-8166-2e6c65bbfea2-kube-api-access-4ls7c\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.113080 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.113079 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91606ba2-0df7-4092-8166-2e6c65bbfea2-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.215004 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.214979 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 23:02:14.262953 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.262920 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 23:02:14.318447 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.318413 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-workload-certs\") pod \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " Oct 29 23:02:14.318625 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.318461 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-workload-socket\") pod \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " Oct 29 23:02:14.318625 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.318538 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-data\") pod \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " Oct 29 23:02:14.318803 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.318669 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-envoy\") pod \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " Oct 29 23:02:14.318803 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.318798 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istiod-ca-cert\") pod \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " Oct 29 23:02:14.318924 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.318851 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-credential-socket\") pod \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " Oct 29 23:02:14.318924 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.318892 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-kube-api-access-ksx9f\" (UniqueName: \"kubernetes.io/projected/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-kube-api-access-ksx9f\") pod \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " Oct 29 23:02:14.318924 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.318920 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-token\" (UniqueName: \"kubernetes.io/projected/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-token\") pod \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " Oct 29 23:02:14.319085 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.318968 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-podinfo\") pod \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\" (UID: \"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01\") " Oct 29 23:02:14.324167 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.324135 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-workload-certs" pod "0c2e7b94-f0cf-424d-a296-c2fe8e60ff01" (UID: "0c2e7b94-f0cf-424d-a296-c2fe8e60ff01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:14.324466 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.324443 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-data" pod "0c2e7b94-f0cf-424d-a296-c2fe8e60ff01" (UID: "0c2e7b94-f0cf-424d-a296-c2fe8e60ff01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:14.324670 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.324649 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-workload-socket" pod "0c2e7b94-f0cf-424d-a296-c2fe8e60ff01" (UID: "0c2e7b94-f0cf-424d-a296-c2fe8e60ff01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:14.324925 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.324905 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istiod-ca-cert" pod "0c2e7b94-f0cf-424d-a296-c2fe8e60ff01" (UID: "0c2e7b94-f0cf-424d-a296-c2fe8e60ff01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:14.325882 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.325850 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-credential-socket" pod "0c2e7b94-f0cf-424d-a296-c2fe8e60ff01" (UID: "0c2e7b94-f0cf-424d-a296-c2fe8e60ff01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:14.326871 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.326843 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-kube-api-access-ksx9f" pod "0c2e7b94-f0cf-424d-a296-c2fe8e60ff01" (UID: "0c2e7b94-f0cf-424d-a296-c2fe8e60ff01"). InnerVolumeSpecName "kube-api-access-ksx9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:14.327703 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.327675 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-podinfo" pod "0c2e7b94-f0cf-424d-a296-c2fe8e60ff01" (UID: "0c2e7b94-f0cf-424d-a296-c2fe8e60ff01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:02:14.328844 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.328818 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-token" pod "0c2e7b94-f0cf-424d-a296-c2fe8e60ff01" (UID: "0c2e7b94-f0cf-424d-a296-c2fe8e60ff01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:14.329554 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.329531 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-envoy" pod "0c2e7b94-f0cf-424d-a296-c2fe8e60ff01" (UID: "0c2e7b94-f0cf-424d-a296-c2fe8e60ff01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:14.333902 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.333883 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 23:02:14.424621 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.424587 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-token\" (UniqueName: \"kubernetes.io/projected/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-token\") pod \"0bb6b077-1785-4673-9dc9-80258dcbecf3\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " Oct 29 23:02:14.424820 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.424653 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-podinfo\") pod \"6103a8bf-2a28-4991-8b55-fecb16515d7f\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " Oct 29 23:02:14.424820 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.424704 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-workload-socket\") pod \"6103a8bf-2a28-4991-8b55-fecb16515d7f\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " Oct 29 23:02:14.424820 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.424733 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-workload-socket\") pod \"0bb6b077-1785-4673-9dc9-80258dcbecf3\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " Oct 29 23:02:14.424820 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.424756 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-workload-certs\") pod \"0bb6b077-1785-4673-9dc9-80258dcbecf3\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " Oct 29 23:02:14.424820 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.424781 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-podinfo\") pod \"0bb6b077-1785-4673-9dc9-80258dcbecf3\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " Oct 29 23:02:14.425119 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.424822 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-data\") pod \"0bb6b077-1785-4673-9dc9-80258dcbecf3\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " Oct 29 23:02:14.425119 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.424850 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-credential-socket\") pod \"6103a8bf-2a28-4991-8b55-fecb16515d7f\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " Oct 29 23:02:14.425119 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.424889 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6103a8bf-2a28-4991-8b55-fecb16515d7f-kube-api-access-xdjr5\" (UniqueName: \"kubernetes.io/projected/6103a8bf-2a28-4991-8b55-fecb16515d7f-kube-api-access-xdjr5\") pod \"6103a8bf-2a28-4991-8b55-fecb16515d7f\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " Oct 29 23:02:14.425119 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.424928 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-token\" (UniqueName: \"kubernetes.io/projected/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-token\") pod \"6103a8bf-2a28-4991-8b55-fecb16515d7f\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " Oct 29 23:02:14.425119 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.424966 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-envoy\") pod \"0bb6b077-1785-4673-9dc9-80258dcbecf3\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " Oct 29 23:02:14.425119 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.425038 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bb6b077-1785-4673-9dc9-80258dcbecf3-kube-api-access-62w66\" (UniqueName: \"kubernetes.io/projected/0bb6b077-1785-4673-9dc9-80258dcbecf3-kube-api-access-62w66\") pod \"0bb6b077-1785-4673-9dc9-80258dcbecf3\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " Oct 29 23:02:14.425538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.425275 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bb6b077-1785-4673-9dc9-80258dcbecf3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb6b077-1785-4673-9dc9-80258dcbecf3-istiod-ca-cert\") pod \"0bb6b077-1785-4673-9dc9-80258dcbecf3\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " Oct 29 23:02:14.425538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.425305 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-credential-socket" pod "6103a8bf-2a28-4991-8b55-fecb16515d7f" (UID: "6103a8bf-2a28-4991-8b55-fecb16515d7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:14.425538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.425346 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-workload-certs\") pod \"6103a8bf-2a28-4991-8b55-fecb16515d7f\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " Oct 29 23:02:14.425538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.425393 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-envoy\") pod \"6103a8bf-2a28-4991-8b55-fecb16515d7f\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " Oct 29 23:02:14.425538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.425423 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6103a8bf-2a28-4991-8b55-fecb16515d7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6103a8bf-2a28-4991-8b55-fecb16515d7f-istiod-ca-cert\") pod \"6103a8bf-2a28-4991-8b55-fecb16515d7f\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " Oct 29 23:02:14.425538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.425470 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-data\") pod \"6103a8bf-2a28-4991-8b55-fecb16515d7f\" (UID: \"6103a8bf-2a28-4991-8b55-fecb16515d7f\") " Oct 29 23:02:14.425538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.425491 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-credential-socket\") pod \"0bb6b077-1785-4673-9dc9-80258dcbecf3\" (UID: \"0bb6b077-1785-4673-9dc9-80258dcbecf3\") " Oct 29 23:02:14.425914 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.425541 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-workload-certs" pod "0bb6b077-1785-4673-9dc9-80258dcbecf3" (UID: "0bb6b077-1785-4673-9dc9-80258dcbecf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:14.425914 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.425688 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-workload-socket" pod "0bb6b077-1785-4673-9dc9-80258dcbecf3" (UID: "0bb6b077-1785-4673-9dc9-80258dcbecf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:14.426568 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.426528 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.426568 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.426559 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.426811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.426577 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.426811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.426591 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.426811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.426605 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.426811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.426619 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.426811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.426633 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksx9f\" (UniqueName: \"kubernetes.io/projected/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-kube-api-access-ksx9f\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.426811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.426648 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.426811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.426661 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.426811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.426674 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.426811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.426689 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.426811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.426701 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.427590 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.426996 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-credential-socket" pod "0bb6b077-1785-4673-9dc9-80258dcbecf3" (UID: "0bb6b077-1785-4673-9dc9-80258dcbecf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:14.427590 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.427375 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb6b077-1785-4673-9dc9-80258dcbecf3-istiod-ca-cert" pod "0bb6b077-1785-4673-9dc9-80258dcbecf3" (UID: "0bb6b077-1785-4673-9dc9-80258dcbecf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:14.428600 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.428325 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-workload-socket" pod "6103a8bf-2a28-4991-8b55-fecb16515d7f" (UID: "6103a8bf-2a28-4991-8b55-fecb16515d7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:14.428600 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.428558 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-data" pod "6103a8bf-2a28-4991-8b55-fecb16515d7f" (UID: "6103a8bf-2a28-4991-8b55-fecb16515d7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:14.428770 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.428743 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-workload-certs" pod "6103a8bf-2a28-4991-8b55-fecb16515d7f" (UID: "6103a8bf-2a28-4991-8b55-fecb16515d7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:14.429583 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.429552 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6103a8bf-2a28-4991-8b55-fecb16515d7f-istiod-ca-cert" pod "6103a8bf-2a28-4991-8b55-fecb16515d7f" (UID: "6103a8bf-2a28-4991-8b55-fecb16515d7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:14.430152 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.430125 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-data" pod "0bb6b077-1785-4673-9dc9-80258dcbecf3" (UID: "0bb6b077-1785-4673-9dc9-80258dcbecf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:14.431396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.430800 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-token" pod "0bb6b077-1785-4673-9dc9-80258dcbecf3" (UID: "0bb6b077-1785-4673-9dc9-80258dcbecf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:14.431627 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.431590 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-podinfo" pod "0bb6b077-1785-4673-9dc9-80258dcbecf3" (UID: "0bb6b077-1785-4673-9dc9-80258dcbecf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:02:14.432387 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.432285 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-envoy" pod "0bb6b077-1785-4673-9dc9-80258dcbecf3" (UID: "0bb6b077-1785-4673-9dc9-80258dcbecf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:14.432387 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.432353 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-envoy" pod "6103a8bf-2a28-4991-8b55-fecb16515d7f" (UID: "6103a8bf-2a28-4991-8b55-fecb16515d7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:14.432387 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.432372 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-podinfo" pod "6103a8bf-2a28-4991-8b55-fecb16515d7f" (UID: "6103a8bf-2a28-4991-8b55-fecb16515d7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:02:14.433070 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.433043 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb6b077-1785-4673-9dc9-80258dcbecf3-kube-api-access-62w66" pod "0bb6b077-1785-4673-9dc9-80258dcbecf3" (UID: "0bb6b077-1785-4673-9dc9-80258dcbecf3"). InnerVolumeSpecName "kube-api-access-62w66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:14.433167 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.433100 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6103a8bf-2a28-4991-8b55-fecb16515d7f-kube-api-access-xdjr5" pod "6103a8bf-2a28-4991-8b55-fecb16515d7f" (UID: "6103a8bf-2a28-4991-8b55-fecb16515d7f"). InnerVolumeSpecName "kube-api-access-xdjr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:14.434142 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.434119 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-token" pod "6103a8bf-2a28-4991-8b55-fecb16515d7f" (UID: "6103a8bf-2a28-4991-8b55-fecb16515d7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:14.527904 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.527818 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb6b077-1785-4673-9dc9-80258dcbecf3-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.527904 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.527846 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.527904 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.527855 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.527904 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.527865 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6103a8bf-2a28-4991-8b55-fecb16515d7f-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.527904 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.527873 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.527904 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.527882 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.527904 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.527892 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.527904 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.527901 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.527904 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.527908 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6103a8bf-2a28-4991-8b55-fecb16515d7f-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.528347 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.527918 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.528347 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.527926 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.528347 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.527934 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdjr5\" (UniqueName: \"kubernetes.io/projected/6103a8bf-2a28-4991-8b55-fecb16515d7f-kube-api-access-xdjr5\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.528347 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.527942 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6103a8bf-2a28-4991-8b55-fecb16515d7f-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.528347 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.527950 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb6b077-1785-4673-9dc9-80258dcbecf3-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.528347 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.527958 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62w66\" (UniqueName: \"kubernetes.io/projected/0bb6b077-1785-4673-9dc9-80258dcbecf3-kube-api-access-62w66\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:14.727591 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.727552 4821 generic.go:356] "Generic (PLEG): container finished" podID="0bb6b077-1785-4673-9dc9-80258dcbecf3" containerID="c56926b65dfa34cafd80fac4c455899fe4bf67e7a20f0cc082eb47b0e159fa6d" exitCode=0 Oct 29 23:02:14.727791 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.727610 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" Oct 29 23:02:14.727791 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.727646 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" event={"ID":"0bb6b077-1785-4673-9dc9-80258dcbecf3","Type":"ContainerDied","Data":"c56926b65dfa34cafd80fac4c455899fe4bf67e7a20f0cc082eb47b0e159fa6d"} Oct 29 23:02:14.727791 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.727697 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42" event={"ID":"0bb6b077-1785-4673-9dc9-80258dcbecf3","Type":"ContainerDied","Data":"30cbd3ed625e2a0b62416d2e3dc36d8e0017c97b07d1761b17316181534660d4"} Oct 29 23:02:14.727791 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.727731 4821 scope.go:122] "RemoveContainer" containerID="c56926b65dfa34cafd80fac4c455899fe4bf67e7a20f0cc082eb47b0e159fa6d" Oct 29 23:02:14.729869 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.729841 4821 generic.go:356] "Generic (PLEG): container finished" podID="0c2e7b94-f0cf-424d-a296-c2fe8e60ff01" containerID="a8c7cdf4c3a29e9f2627730816b381e54a3446275af17a1f0f7fc7f0d4ffbf31" exitCode=0 Oct 29 23:02:14.730001 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.729874 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" event={"ID":"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01","Type":"ContainerDied","Data":"a8c7cdf4c3a29e9f2627730816b381e54a3446275af17a1f0f7fc7f0d4ffbf31"} Oct 29 23:02:14.730001 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.729910 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" Oct 29 23:02:14.730001 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.729927 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh" event={"ID":"0c2e7b94-f0cf-424d-a296-c2fe8e60ff01","Type":"ContainerDied","Data":"7a85f7e8a989e2db21566b6efef32b6a27efe7cb04c486a082d343c059371eea"} Oct 29 23:02:14.731831 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.731806 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w" Oct 29 23:02:14.731956 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.731838 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w" event={"ID":"91606ba2-0df7-4092-8166-2e6c65bbfea2","Type":"ContainerDied","Data":"95353ac18faa6e8392b01d261e0c07abdcf1919f90170f79562fe251d9b63e66"} Oct 29 23:02:14.731956 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.731891 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95353ac18faa6e8392b01d261e0c07abdcf1919f90170f79562fe251d9b63e66" Oct 29 23:02:14.733484 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.733464 4821 generic.go:356] "Generic (PLEG): container finished" podID="6103a8bf-2a28-4991-8b55-fecb16515d7f" containerID="23cf050765136816912dd67848761598c1b3143d1d17aa6109ba0e68ae56ec19" exitCode=0 Oct 29 23:02:14.733567 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.733483 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" event={"ID":"6103a8bf-2a28-4991-8b55-fecb16515d7f","Type":"ContainerDied","Data":"23cf050765136816912dd67848761598c1b3143d1d17aa6109ba0e68ae56ec19"} Oct 29 23:02:14.733567 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.733516 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" Oct 29 23:02:14.733682 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.733589 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn" event={"ID":"6103a8bf-2a28-4991-8b55-fecb16515d7f","Type":"ContainerDied","Data":"f8057c1c4dd5fea6a01404c2b672930a79da296c60055b5d7ff5b83ad0747a85"} Oct 29 23:02:14.741087 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.741072 4821 scope.go:122] "RemoveContainer" containerID="c56926b65dfa34cafd80fac4c455899fe4bf67e7a20f0cc082eb47b0e159fa6d" Oct 29 23:02:14.741532 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:02:14.741379 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c56926b65dfa34cafd80fac4c455899fe4bf67e7a20f0cc082eb47b0e159fa6d\": container with ID starting with c56926b65dfa34cafd80fac4c455899fe4bf67e7a20f0cc082eb47b0e159fa6d not found: ID does not exist" containerID="c56926b65dfa34cafd80fac4c455899fe4bf67e7a20f0cc082eb47b0e159fa6d" Oct 29 23:02:14.741532 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.741410 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c56926b65dfa34cafd80fac4c455899fe4bf67e7a20f0cc082eb47b0e159fa6d"} err="failed to get container status \"c56926b65dfa34cafd80fac4c455899fe4bf67e7a20f0cc082eb47b0e159fa6d\": rpc error: code = NotFound desc = could not find container \"c56926b65dfa34cafd80fac4c455899fe4bf67e7a20f0cc082eb47b0e159fa6d\": container with ID starting with c56926b65dfa34cafd80fac4c455899fe4bf67e7a20f0cc082eb47b0e159fa6d not found: ID does not exist" Oct 29 23:02:14.741532 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.741435 4821 scope.go:122] "RemoveContainer" containerID="a8c7cdf4c3a29e9f2627730816b381e54a3446275af17a1f0f7fc7f0d4ffbf31" Oct 29 23:02:14.753934 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.753910 4821 scope.go:122] "RemoveContainer" containerID="a8c7cdf4c3a29e9f2627730816b381e54a3446275af17a1f0f7fc7f0d4ffbf31" Oct 29 23:02:14.754816 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:02:14.754794 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c7cdf4c3a29e9f2627730816b381e54a3446275af17a1f0f7fc7f0d4ffbf31\": container with ID starting with a8c7cdf4c3a29e9f2627730816b381e54a3446275af17a1f0f7fc7f0d4ffbf31 not found: ID does not exist" containerID="a8c7cdf4c3a29e9f2627730816b381e54a3446275af17a1f0f7fc7f0d4ffbf31" Oct 29 23:02:14.754922 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.754824 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c7cdf4c3a29e9f2627730816b381e54a3446275af17a1f0f7fc7f0d4ffbf31"} err="failed to get container status \"a8c7cdf4c3a29e9f2627730816b381e54a3446275af17a1f0f7fc7f0d4ffbf31\": rpc error: code = NotFound desc = could not find container \"a8c7cdf4c3a29e9f2627730816b381e54a3446275af17a1f0f7fc7f0d4ffbf31\": container with ID starting with a8c7cdf4c3a29e9f2627730816b381e54a3446275af17a1f0f7fc7f0d4ffbf31 not found: ID does not exist" Oct 29 23:02:14.754922 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.754847 4821 scope.go:122] "RemoveContainer" containerID="23cf050765136816912dd67848761598c1b3143d1d17aa6109ba0e68ae56ec19" Oct 29 23:02:14.764785 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.764770 4821 scope.go:122] "RemoveContainer" containerID="23cf050765136816912dd67848761598c1b3143d1d17aa6109ba0e68ae56ec19" Oct 29 23:02:14.765073 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:02:14.765053 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23cf050765136816912dd67848761598c1b3143d1d17aa6109ba0e68ae56ec19\": container with ID starting with 23cf050765136816912dd67848761598c1b3143d1d17aa6109ba0e68ae56ec19 not found: ID does not exist" containerID="23cf050765136816912dd67848761598c1b3143d1d17aa6109ba0e68ae56ec19" Oct 29 23:02:14.765128 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.765093 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23cf050765136816912dd67848761598c1b3143d1d17aa6109ba0e68ae56ec19"} err="failed to get container status \"23cf050765136816912dd67848761598c1b3143d1d17aa6109ba0e68ae56ec19\": rpc error: code = NotFound desc = could not find container \"23cf050765136816912dd67848761598c1b3143d1d17aa6109ba0e68ae56ec19\": container with ID starting with 23cf050765136816912dd67848761598c1b3143d1d17aa6109ba0e68ae56ec19 not found: ID does not exist" Oct 29 23:02:14.818493 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.818429 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42"] Oct 29 23:02:14.832507 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.832483 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-swz42"] Oct 29 23:02:14.859791 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.859759 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh"] Oct 29 23:02:14.863708 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.863604 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-cjxzh"] Oct 29 23:02:14.904036 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.904005 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn"] Oct 29 23:02:14.907880 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:14.907854 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-svhmn"] Oct 29 23:02:16.171335 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:16.171302 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bb6b077-1785-4673-9dc9-80258dcbecf3" path="/var/lib/kubelet/pods/0bb6b077-1785-4673-9dc9-80258dcbecf3/volumes" Oct 29 23:02:16.171848 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:16.171832 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c2e7b94-f0cf-424d-a296-c2fe8e60ff01" path="/var/lib/kubelet/pods/0c2e7b94-f0cf-424d-a296-c2fe8e60ff01/volumes" Oct 29 23:02:16.172287 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:16.172273 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6103a8bf-2a28-4991-8b55-fecb16515d7f" path="/var/lib/kubelet/pods/6103a8bf-2a28-4991-8b55-fecb16515d7f/volumes" Oct 29 23:02:19.355046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:19.355007 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n"] Oct 29 23:02:19.355532 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:19.355281 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n" podUID="650729e2-7050-4990-b7f5-ec4fa4c51947" containerName="maas-api" containerID="cri-o://c43c9e058ffb72c1a0d7396903a814e8e81cb154f3940fb3dd0dd845c0435f5e" gracePeriod=30 Oct 29 23:02:19.788729 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:19.788421 4821 generic.go:356] "Generic (PLEG): container finished" podID="650729e2-7050-4990-b7f5-ec4fa4c51947" containerID="c43c9e058ffb72c1a0d7396903a814e8e81cb154f3940fb3dd0dd845c0435f5e" exitCode=0 Oct 29 23:02:19.788729 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:19.788585 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n" event={"ID":"650729e2-7050-4990-b7f5-ec4fa4c51947","Type":"ContainerDied","Data":"c43c9e058ffb72c1a0d7396903a814e8e81cb154f3940fb3dd0dd845c0435f5e"} Oct 29 23:02:19.788729 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:19.788677 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n" event={"ID":"650729e2-7050-4990-b7f5-ec4fa4c51947","Type":"ContainerDied","Data":"310e71b37812f5f9e160c606702eaedff017aea546c93ec742a41946484e711a"} Oct 29 23:02:19.788729 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:19.788703 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="310e71b37812f5f9e160c606702eaedff017aea546c93ec742a41946484e711a" Oct 29 23:02:19.812225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:19.799396 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n" Oct 29 23:02:19.840225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:19.839899 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/650729e2-7050-4990-b7f5-ec4fa4c51947-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/650729e2-7050-4990-b7f5-ec4fa4c51947-openshift-service-ca\") pod \"650729e2-7050-4990-b7f5-ec4fa4c51947\" (UID: \"650729e2-7050-4990-b7f5-ec4fa4c51947\") " Oct 29 23:02:19.840225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:19.839951 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/650729e2-7050-4990-b7f5-ec4fa4c51947-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/650729e2-7050-4990-b7f5-ec4fa4c51947-maas-api-tls\") pod \"650729e2-7050-4990-b7f5-ec4fa4c51947\" (UID: \"650729e2-7050-4990-b7f5-ec4fa4c51947\") " Oct 29 23:02:19.840225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:19.840157 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/650729e2-7050-4990-b7f5-ec4fa4c51947-kube-api-access-kgqxf\" (UniqueName: \"kubernetes.io/projected/650729e2-7050-4990-b7f5-ec4fa4c51947-kube-api-access-kgqxf\") pod \"650729e2-7050-4990-b7f5-ec4fa4c51947\" (UID: \"650729e2-7050-4990-b7f5-ec4fa4c51947\") " Oct 29 23:02:19.840606 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:19.840380 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/650729e2-7050-4990-b7f5-ec4fa4c51947-metrics-certs\" (UniqueName: \"kubernetes.io/secret/650729e2-7050-4990-b7f5-ec4fa4c51947-metrics-certs\") pod \"650729e2-7050-4990-b7f5-ec4fa4c51947\" (UID: \"650729e2-7050-4990-b7f5-ec4fa4c51947\") " Oct 29 23:02:19.853263 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:19.844404 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/650729e2-7050-4990-b7f5-ec4fa4c51947-openshift-service-ca" pod "650729e2-7050-4990-b7f5-ec4fa4c51947" (UID: "650729e2-7050-4990-b7f5-ec4fa4c51947"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:19.946251 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:19.942385 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/650729e2-7050-4990-b7f5-ec4fa4c51947-metrics-certs" pod "650729e2-7050-4990-b7f5-ec4fa4c51947" (UID: "650729e2-7050-4990-b7f5-ec4fa4c51947"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:02:19.946251 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:19.943905 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/650729e2-7050-4990-b7f5-ec4fa4c51947-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.946251 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:19.943928 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/650729e2-7050-4990-b7f5-ec4fa4c51947-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:19.994944 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:19.994394 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/650729e2-7050-4990-b7f5-ec4fa4c51947-maas-api-tls" pod "650729e2-7050-4990-b7f5-ec4fa4c51947" (UID: "650729e2-7050-4990-b7f5-ec4fa4c51947"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:02:19.997666 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:19.997627 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/650729e2-7050-4990-b7f5-ec4fa4c51947-kube-api-access-kgqxf" pod "650729e2-7050-4990-b7f5-ec4fa4c51947" (UID: "650729e2-7050-4990-b7f5-ec4fa4c51947"). InnerVolumeSpecName "kube-api-access-kgqxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:20.045311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:20.045275 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgqxf\" (UniqueName: \"kubernetes.io/projected/650729e2-7050-4990-b7f5-ec4fa4c51947-kube-api-access-kgqxf\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:20.045311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:20.045307 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/650729e2-7050-4990-b7f5-ec4fa4c51947-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:20.263347 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:20.263185 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r"] Oct 29 23:02:20.267075 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:20.267050 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3e268-s9f5r"] Oct 29 23:02:20.796285 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:20.796255 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n" Oct 29 23:02:20.852957 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:20.852922 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n"] Oct 29 23:02:20.866269 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:20.866239 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d3e268-854578f95f-z7n8n"] Oct 29 23:02:21.998739 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:21.998709 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm"] Oct 29 23:02:21.999262 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:21.998948 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" podUID="03949775-cc5e-4c51-8179-f53230828245" containerName="maas-api" containerID="cri-o://04b46a3472fa5f1a74e1a7a158c3a7e0e288222bbaa7952f0e874e3772ce6947" gracePeriod=30 Oct 29 23:02:22.175700 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:22.175665 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1299a73b-9d42-44dc-aa4c-e535358ba2c5" path="/var/lib/kubelet/pods/1299a73b-9d42-44dc-aa4c-e535358ba2c5/volumes" Oct 29 23:02:22.176300 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:22.176279 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="650729e2-7050-4990-b7f5-ec4fa4c51947" path="/var/lib/kubelet/pods/650729e2-7050-4990-b7f5-ec4fa4c51947/volumes" Oct 29 23:02:22.316105 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:22.316076 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" Oct 29 23:02:22.391314 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:22.385876 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03949775-cc5e-4c51-8179-f53230828245-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03949775-cc5e-4c51-8179-f53230828245-maas-api-tls\") pod \"03949775-cc5e-4c51-8179-f53230828245\" (UID: \"03949775-cc5e-4c51-8179-f53230828245\") " Oct 29 23:02:22.391314 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:22.385962 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03949775-cc5e-4c51-8179-f53230828245-metrics-certs\" (UniqueName: \"kubernetes.io/secret/03949775-cc5e-4c51-8179-f53230828245-metrics-certs\") pod \"03949775-cc5e-4c51-8179-f53230828245\" (UID: \"03949775-cc5e-4c51-8179-f53230828245\") " Oct 29 23:02:22.391314 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:22.386018 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03949775-cc5e-4c51-8179-f53230828245-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03949775-cc5e-4c51-8179-f53230828245-openshift-service-ca\") pod \"03949775-cc5e-4c51-8179-f53230828245\" (UID: \"03949775-cc5e-4c51-8179-f53230828245\") " Oct 29 23:02:22.391314 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:22.386121 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03949775-cc5e-4c51-8179-f53230828245-kube-api-access-m9x8z\" (UniqueName: \"kubernetes.io/projected/03949775-cc5e-4c51-8179-f53230828245-kube-api-access-m9x8z\") pod \"03949775-cc5e-4c51-8179-f53230828245\" (UID: \"03949775-cc5e-4c51-8179-f53230828245\") " Oct 29 23:02:22.398641 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:22.398090 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03949775-cc5e-4c51-8179-f53230828245-kube-api-access-m9x8z" pod "03949775-cc5e-4c51-8179-f53230828245" (UID: "03949775-cc5e-4c51-8179-f53230828245"). InnerVolumeSpecName "kube-api-access-m9x8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:22.408273 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:22.407084 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03949775-cc5e-4c51-8179-f53230828245-openshift-service-ca" pod "03949775-cc5e-4c51-8179-f53230828245" (UID: "03949775-cc5e-4c51-8179-f53230828245"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:22.425278 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:22.425188 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03949775-cc5e-4c51-8179-f53230828245-metrics-certs" pod "03949775-cc5e-4c51-8179-f53230828245" (UID: "03949775-cc5e-4c51-8179-f53230828245"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:02:22.462529 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:22.462462 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03949775-cc5e-4c51-8179-f53230828245-maas-api-tls" pod "03949775-cc5e-4c51-8179-f53230828245" (UID: "03949775-cc5e-4c51-8179-f53230828245"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:02:22.491942 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:22.490351 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03949775-cc5e-4c51-8179-f53230828245-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:22.491942 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:22.490384 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03949775-cc5e-4c51-8179-f53230828245-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:22.491942 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:22.490398 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03949775-cc5e-4c51-8179-f53230828245-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:22.491942 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:22.490419 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9x8z\" (UniqueName: \"kubernetes.io/projected/03949775-cc5e-4c51-8179-f53230828245-kube-api-access-m9x8z\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:22.825183 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:22.825141 4821 generic.go:356] "Generic (PLEG): container finished" podID="03949775-cc5e-4c51-8179-f53230828245" containerID="04b46a3472fa5f1a74e1a7a158c3a7e0e288222bbaa7952f0e874e3772ce6947" exitCode=0 Oct 29 23:02:22.825404 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:22.825247 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" Oct 29 23:02:22.825404 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:22.825304 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" event={"ID":"03949775-cc5e-4c51-8179-f53230828245","Type":"ContainerDied","Data":"04b46a3472fa5f1a74e1a7a158c3a7e0e288222bbaa7952f0e874e3772ce6947"} Oct 29 23:02:22.825404 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:22.825376 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm" event={"ID":"03949775-cc5e-4c51-8179-f53230828245","Type":"ContainerDied","Data":"5bbc9941339953533a98a2dcaca9a24bd52f3fe8e1fc9365a6f1c504a3b4f4ce"} Oct 29 23:02:22.825404 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:22.825397 4821 scope.go:122] "RemoveContainer" containerID="04b46a3472fa5f1a74e1a7a158c3a7e0e288222bbaa7952f0e874e3772ce6947" Oct 29 23:02:22.843074 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:22.843058 4821 scope.go:122] "RemoveContainer" containerID="04b46a3472fa5f1a74e1a7a158c3a7e0e288222bbaa7952f0e874e3772ce6947" Oct 29 23:02:22.843391 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:02:22.843367 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04b46a3472fa5f1a74e1a7a158c3a7e0e288222bbaa7952f0e874e3772ce6947\": container with ID starting with 04b46a3472fa5f1a74e1a7a158c3a7e0e288222bbaa7952f0e874e3772ce6947 not found: ID does not exist" containerID="04b46a3472fa5f1a74e1a7a158c3a7e0e288222bbaa7952f0e874e3772ce6947" Oct 29 23:02:22.843483 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:22.843402 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04b46a3472fa5f1a74e1a7a158c3a7e0e288222bbaa7952f0e874e3772ce6947"} err="failed to get container status \"04b46a3472fa5f1a74e1a7a158c3a7e0e288222bbaa7952f0e874e3772ce6947\": rpc error: code = NotFound desc = could not find container \"04b46a3472fa5f1a74e1a7a158c3a7e0e288222bbaa7952f0e874e3772ce6947\": container with ID starting with 04b46a3472fa5f1a74e1a7a158c3a7e0e288222bbaa7952f0e874e3772ce6947 not found: ID does not exist" Oct 29 23:02:22.883107 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:22.883076 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm"] Oct 29 23:02:22.887961 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:22.887933 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-49dbb5-7fd9f47969-92srm"] Oct 29 23:02:23.597338 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:23.597304 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w"] Oct 29 23:02:23.609837 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:23.609806 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-13aa41-xth8w"] Oct 29 23:02:23.893708 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:02:23.893667 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:02:23.973327 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:23.973287 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj"] Oct 29 23:02:23.973525 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:23.973491 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" podUID="240bea83-173e-4b4a-9989-9608028afac4" containerName="istio-proxy" containerID="cri-o://bce99867a5f93185cc58d3235746788a460b32bc2e8ee471e53bf72856f6eb2f" gracePeriod=30 Oct 29 23:02:23.977362 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:23.977337 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg"] Oct 29 23:02:23.977589 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:23.977550 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" podUID="68d7791f-31f4-4552-91d0-120f640ad1fa" containerName="istio-proxy" containerID="cri-o://3d63bf0b80f83ed78e7cc83b12a8ca66f5b207ad0b5b44098d3d9531da1ed3bd" gracePeriod=30 Oct 29 23:02:23.988390 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:23.988360 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq"] Oct 29 23:02:23.988606 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:23.988572 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" podUID="3f99e4c4-51fa-4e77-b863-c8867c9a08c2" containerName="istio-proxy" containerID="cri-o://7dba42270b7085c57d0ab469137137f199031bbc0e964f1fbc924dd0c21834c2" gracePeriod=30 Oct 29 23:02:24.171639 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:24.171563 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03949775-cc5e-4c51-8179-f53230828245" path="/var/lib/kubelet/pods/03949775-cc5e-4c51-8179-f53230828245/volumes" Oct 29 23:02:24.172073 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:24.172058 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91606ba2-0df7-4092-8166-2e6c65bbfea2" path="/var/lib/kubelet/pods/91606ba2-0df7-4092-8166-2e6c65bbfea2/volumes" Oct 29 23:02:26.605605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:26.605575 4821 scope.go:122] "RemoveContainer" containerID="b537fac20fec3d960c92a6feea96feb58f2e12462ae08191f3cabec64409b3a3" Oct 29 23:02:29.537736 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.537458 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 23:02:29.551748 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.551720 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-credential-socket\") pod \"68d7791f-31f4-4552-91d0-120f640ad1fa\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " Oct 29 23:02:29.551882 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.551765 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-data\") pod \"68d7791f-31f4-4552-91d0-120f640ad1fa\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " Oct 29 23:02:29.551882 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.551817 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-podinfo\") pod \"68d7791f-31f4-4552-91d0-120f640ad1fa\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " Oct 29 23:02:29.552000 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.551897 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68d7791f-31f4-4552-91d0-120f640ad1fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d7791f-31f4-4552-91d0-120f640ad1fa-istiod-ca-cert\") pod \"68d7791f-31f4-4552-91d0-120f640ad1fa\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " Oct 29 23:02:29.552000 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.551990 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-token\" (UniqueName: \"kubernetes.io/projected/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-token\") pod \"68d7791f-31f4-4552-91d0-120f640ad1fa\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " Oct 29 23:02:29.552112 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.552015 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-credential-socket" pod "68d7791f-31f4-4552-91d0-120f640ad1fa" (UID: "68d7791f-31f4-4552-91d0-120f640ad1fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:29.552112 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.552039 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d7791f-31f4-4552-91d0-120f640ad1fa-kube-api-access-2n9m5\" (UniqueName: \"kubernetes.io/projected/68d7791f-31f4-4552-91d0-120f640ad1fa-kube-api-access-2n9m5\") pod \"68d7791f-31f4-4552-91d0-120f640ad1fa\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " Oct 29 23:02:29.552112 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.552071 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-envoy\") pod \"68d7791f-31f4-4552-91d0-120f640ad1fa\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " Oct 29 23:02:29.552338 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.552103 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-data" pod "68d7791f-31f4-4552-91d0-120f640ad1fa" (UID: "68d7791f-31f4-4552-91d0-120f640ad1fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:29.552338 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.552132 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-workload-socket\") pod \"68d7791f-31f4-4552-91d0-120f640ad1fa\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " Oct 29 23:02:29.552338 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.552183 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-workload-certs\") pod \"68d7791f-31f4-4552-91d0-120f640ad1fa\" (UID: \"68d7791f-31f4-4552-91d0-120f640ad1fa\") " Oct 29 23:02:29.552511 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.552359 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d7791f-31f4-4552-91d0-120f640ad1fa-istiod-ca-cert" pod "68d7791f-31f4-4552-91d0-120f640ad1fa" (UID: "68d7791f-31f4-4552-91d0-120f640ad1fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:29.552983 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.552957 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d7791f-31f4-4552-91d0-120f640ad1fa-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.553090 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.552983 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-workload-certs" pod "68d7791f-31f4-4552-91d0-120f640ad1fa" (UID: "68d7791f-31f4-4552-91d0-120f640ad1fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:29.553090 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.552870 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-workload-socket" pod "68d7791f-31f4-4552-91d0-120f640ad1fa" (UID: "68d7791f-31f4-4552-91d0-120f640ad1fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:29.553090 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.552988 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.553090 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.553032 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.555299 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.555270 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-envoy" pod "68d7791f-31f4-4552-91d0-120f640ad1fa" (UID: "68d7791f-31f4-4552-91d0-120f640ad1fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:29.555457 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.555430 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-token" pod "68d7791f-31f4-4552-91d0-120f640ad1fa" (UID: "68d7791f-31f4-4552-91d0-120f640ad1fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:29.555580 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.555478 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d7791f-31f4-4552-91d0-120f640ad1fa-kube-api-access-2n9m5" pod "68d7791f-31f4-4552-91d0-120f640ad1fa" (UID: "68d7791f-31f4-4552-91d0-120f640ad1fa"). InnerVolumeSpecName "kube-api-access-2n9m5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:29.559742 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.559706 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-podinfo" pod "68d7791f-31f4-4552-91d0-120f640ad1fa" (UID: "68d7791f-31f4-4552-91d0-120f640ad1fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:02:29.651041 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.651010 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 23:02:29.654435 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.654411 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-workload-certs\") pod \"240bea83-173e-4b4a-9989-9608028afac4\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " Oct 29 23:02:29.654571 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.654460 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/240bea83-173e-4b4a-9989-9608028afac4-istio-token\" (UniqueName: \"kubernetes.io/projected/240bea83-173e-4b4a-9989-9608028afac4-istio-token\") pod \"240bea83-173e-4b4a-9989-9608028afac4\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " Oct 29 23:02:29.654571 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.654490 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/240bea83-173e-4b4a-9989-9608028afac4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/240bea83-173e-4b4a-9989-9608028afac4-istiod-ca-cert\") pod \"240bea83-173e-4b4a-9989-9608028afac4\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " Oct 29 23:02:29.654696 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.654595 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-workload-socket\") pod \"240bea83-173e-4b4a-9989-9608028afac4\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " Oct 29 23:02:29.654696 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.654643 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/240bea83-173e-4b4a-9989-9608028afac4-kube-api-access-pt5v8\" (UniqueName: \"kubernetes.io/projected/240bea83-173e-4b4a-9989-9608028afac4-kube-api-access-pt5v8\") pod \"240bea83-173e-4b4a-9989-9608028afac4\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " Oct 29 23:02:29.654811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.654720 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-credential-socket\") pod \"240bea83-173e-4b4a-9989-9608028afac4\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " Oct 29 23:02:29.654811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.654734 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-workload-certs" pod "240bea83-173e-4b4a-9989-9608028afac4" (UID: "240bea83-173e-4b4a-9989-9608028afac4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:29.654918 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.654842 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/240bea83-173e-4b4a-9989-9608028afac4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/240bea83-173e-4b4a-9989-9608028afac4-istio-podinfo\") pod \"240bea83-173e-4b4a-9989-9608028afac4\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " Oct 29 23:02:29.654918 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.654855 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/240bea83-173e-4b4a-9989-9608028afac4-istiod-ca-cert" pod "240bea83-173e-4b4a-9989-9608028afac4" (UID: "240bea83-173e-4b4a-9989-9608028afac4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:29.654918 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.654906 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-istio-data\") pod \"240bea83-173e-4b4a-9989-9608028afac4\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " Oct 29 23:02:29.655081 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.654968 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-istio-envoy\") pod \"240bea83-173e-4b4a-9989-9608028afac4\" (UID: \"240bea83-173e-4b4a-9989-9608028afac4\") " Oct 29 23:02:29.655139 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.655075 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-credential-socket" pod "240bea83-173e-4b4a-9989-9608028afac4" (UID: "240bea83-173e-4b4a-9989-9608028afac4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:29.655463 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.655436 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-workload-socket" pod "240bea83-173e-4b4a-9989-9608028afac4" (UID: "240bea83-173e-4b4a-9989-9608028afac4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:29.655575 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.655459 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-istio-data" pod "240bea83-173e-4b4a-9989-9608028afac4" (UID: "240bea83-173e-4b4a-9989-9608028afac4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:29.655658 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.655633 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 23:02:29.655896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.655871 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.655969 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.655901 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.655969 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.655917 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/240bea83-173e-4b4a-9989-9608028afac4-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.655969 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.655931 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.655969 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.655944 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.655969 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.655958 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n9m5\" (UniqueName: \"kubernetes.io/projected/68d7791f-31f4-4552-91d0-120f640ad1fa-kube-api-access-2n9m5\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.656284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.655971 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.656284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.655986 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.656284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.655999 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.656284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.656012 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d7791f-31f4-4552-91d0-120f640ad1fa-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.656284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.656024 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.657085 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.657063 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/240bea83-173e-4b4a-9989-9608028afac4-istio-token" pod "240bea83-173e-4b4a-9989-9608028afac4" (UID: "240bea83-173e-4b4a-9989-9608028afac4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:29.657885 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.657856 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/240bea83-173e-4b4a-9989-9608028afac4-kube-api-access-pt5v8" pod "240bea83-173e-4b4a-9989-9608028afac4" (UID: "240bea83-173e-4b4a-9989-9608028afac4"). InnerVolumeSpecName "kube-api-access-pt5v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:29.658642 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.658611 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/240bea83-173e-4b4a-9989-9608028afac4-istio-podinfo" pod "240bea83-173e-4b4a-9989-9608028afac4" (UID: "240bea83-173e-4b4a-9989-9608028afac4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:02:29.659247 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.659220 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-istio-envoy" pod "240bea83-173e-4b4a-9989-9608028afac4" (UID: "240bea83-173e-4b4a-9989-9608028afac4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:29.757400 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.757309 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-podinfo\") pod \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " Oct 29 23:02:29.757571 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.757404 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-kube-api-access-dsmvg\" (UniqueName: \"kubernetes.io/projected/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-kube-api-access-dsmvg\") pod \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " Oct 29 23:02:29.757571 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.757493 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-credential-socket\") pod \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " Oct 29 23:02:29.757571 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.757520 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-token\" (UniqueName: \"kubernetes.io/projected/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-token\") pod \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " Oct 29 23:02:29.757757 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.757603 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-workload-socket\") pod \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " Oct 29 23:02:29.757757 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.757727 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-workload-certs\") pod \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " Oct 29 23:02:29.757875 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.757746 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-credential-socket" pod "3f99e4c4-51fa-4e77-b863-c8867c9a08c2" (UID: "3f99e4c4-51fa-4e77-b863-c8867c9a08c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:29.757875 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.757859 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istiod-ca-cert\") pod \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " Oct 29 23:02:29.758005 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.757983 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-envoy\") pod \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " Oct 29 23:02:29.758104 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.758031 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-workload-socket" pod "3f99e4c4-51fa-4e77-b863-c8867c9a08c2" (UID: "3f99e4c4-51fa-4e77-b863-c8867c9a08c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:29.758104 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.758047 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-workload-certs" pod "3f99e4c4-51fa-4e77-b863-c8867c9a08c2" (UID: "3f99e4c4-51fa-4e77-b863-c8867c9a08c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:29.758239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.758106 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-data\") pod \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\" (UID: \"3f99e4c4-51fa-4e77-b863-c8867c9a08c2\") " Oct 29 23:02:29.758381 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.758347 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-data" pod "3f99e4c4-51fa-4e77-b863-c8867c9a08c2" (UID: "3f99e4c4-51fa-4e77-b863-c8867c9a08c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:29.758709 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.758669 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istiod-ca-cert" pod "3f99e4c4-51fa-4e77-b863-c8867c9a08c2" (UID: "3f99e4c4-51fa-4e77-b863-c8867c9a08c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:29.759086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.758998 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.759188 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.759096 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/240bea83-173e-4b4a-9989-9608028afac4-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.759188 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.759111 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.759188 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.759124 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.759188 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.759140 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt5v8\" (UniqueName: \"kubernetes.io/projected/240bea83-173e-4b4a-9989-9608028afac4-kube-api-access-pt5v8\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.759188 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.759154 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.759188 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.759168 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/240bea83-173e-4b4a-9989-9608028afac4-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.759188 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.759184 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.759602 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.759218 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/240bea83-173e-4b4a-9989-9608028afac4-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.760252 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.760169 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-podinfo" pod "3f99e4c4-51fa-4e77-b863-c8867c9a08c2" (UID: "3f99e4c4-51fa-4e77-b863-c8867c9a08c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:02:29.760526 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.760506 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-token" pod "3f99e4c4-51fa-4e77-b863-c8867c9a08c2" (UID: "3f99e4c4-51fa-4e77-b863-c8867c9a08c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:29.761062 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.761043 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-kube-api-access-dsmvg" pod "3f99e4c4-51fa-4e77-b863-c8867c9a08c2" (UID: "3f99e4c4-51fa-4e77-b863-c8867c9a08c2"). InnerVolumeSpecName "kube-api-access-dsmvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:29.761515 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.761494 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-envoy" pod "3f99e4c4-51fa-4e77-b863-c8867c9a08c2" (UID: "3f99e4c4-51fa-4e77-b863-c8867c9a08c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:29.861044 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.861004 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.861044 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.861036 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.861044 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.861048 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsmvg\" (UniqueName: \"kubernetes.io/projected/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-kube-api-access-dsmvg\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.861310 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.861060 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f99e4c4-51fa-4e77-b863-c8867c9a08c2-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:29.926102 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.926062 4821 generic.go:356] "Generic (PLEG): container finished" podID="3f99e4c4-51fa-4e77-b863-c8867c9a08c2" containerID="7dba42270b7085c57d0ab469137137f199031bbc0e964f1fbc924dd0c21834c2" exitCode=0 Oct 29 23:02:29.926102 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.926112 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" Oct 29 23:02:29.926387 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.926143 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" event={"ID":"3f99e4c4-51fa-4e77-b863-c8867c9a08c2","Type":"ContainerDied","Data":"7dba42270b7085c57d0ab469137137f199031bbc0e964f1fbc924dd0c21834c2"} Oct 29 23:02:29.926387 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.926193 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq" event={"ID":"3f99e4c4-51fa-4e77-b863-c8867c9a08c2","Type":"ContainerDied","Data":"8a8f66b1f2faf32dfcae36cf0fabda08d958004929f1e6ddef7af09540f4beac"} Oct 29 23:02:29.926387 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.926244 4821 scope.go:122] "RemoveContainer" containerID="7dba42270b7085c57d0ab469137137f199031bbc0e964f1fbc924dd0c21834c2" Oct 29 23:02:29.927774 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.927679 4821 generic.go:356] "Generic (PLEG): container finished" podID="240bea83-173e-4b4a-9989-9608028afac4" containerID="bce99867a5f93185cc58d3235746788a460b32bc2e8ee471e53bf72856f6eb2f" exitCode=0 Oct 29 23:02:29.927774 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.927736 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" Oct 29 23:02:29.927774 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.927768 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" event={"ID":"240bea83-173e-4b4a-9989-9608028afac4","Type":"ContainerDied","Data":"bce99867a5f93185cc58d3235746788a460b32bc2e8ee471e53bf72856f6eb2f"} Oct 29 23:02:29.927774 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.927820 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj" event={"ID":"240bea83-173e-4b4a-9989-9608028afac4","Type":"ContainerDied","Data":"4e1284d036cfba76a80582169854337fa14bdfcd626178e37c79b277d802e1b3"} Oct 29 23:02:29.929569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.929543 4821 generic.go:356] "Generic (PLEG): container finished" podID="68d7791f-31f4-4552-91d0-120f640ad1fa" containerID="3d63bf0b80f83ed78e7cc83b12a8ca66f5b207ad0b5b44098d3d9531da1ed3bd" exitCode=0 Oct 29 23:02:29.929698 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.929595 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" Oct 29 23:02:29.929698 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.929624 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" event={"ID":"68d7791f-31f4-4552-91d0-120f640ad1fa","Type":"ContainerDied","Data":"3d63bf0b80f83ed78e7cc83b12a8ca66f5b207ad0b5b44098d3d9531da1ed3bd"} Oct 29 23:02:29.929698 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.929662 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg" event={"ID":"68d7791f-31f4-4552-91d0-120f640ad1fa","Type":"ContainerDied","Data":"167aa22c3fc2510e010c60a7b1737f2a4f47ec3e857f3f5d121bf9532827a2fc"} Oct 29 23:02:29.938002 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.937981 4821 scope.go:122] "RemoveContainer" containerID="7dba42270b7085c57d0ab469137137f199031bbc0e964f1fbc924dd0c21834c2" Oct 29 23:02:29.938382 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:02:29.938349 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dba42270b7085c57d0ab469137137f199031bbc0e964f1fbc924dd0c21834c2\": container with ID starting with 7dba42270b7085c57d0ab469137137f199031bbc0e964f1fbc924dd0c21834c2 not found: ID does not exist" containerID="7dba42270b7085c57d0ab469137137f199031bbc0e964f1fbc924dd0c21834c2" Oct 29 23:02:29.938517 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.938389 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dba42270b7085c57d0ab469137137f199031bbc0e964f1fbc924dd0c21834c2"} err="failed to get container status \"7dba42270b7085c57d0ab469137137f199031bbc0e964f1fbc924dd0c21834c2\": rpc error: code = NotFound desc = could not find container \"7dba42270b7085c57d0ab469137137f199031bbc0e964f1fbc924dd0c21834c2\": container with ID starting with 7dba42270b7085c57d0ab469137137f199031bbc0e964f1fbc924dd0c21834c2 not found: ID does not exist" Oct 29 23:02:29.938517 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.938412 4821 scope.go:122] "RemoveContainer" containerID="bce99867a5f93185cc58d3235746788a460b32bc2e8ee471e53bf72856f6eb2f" Oct 29 23:02:29.948757 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.948736 4821 scope.go:122] "RemoveContainer" containerID="bce99867a5f93185cc58d3235746788a460b32bc2e8ee471e53bf72856f6eb2f" Oct 29 23:02:29.949066 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:02:29.949048 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce99867a5f93185cc58d3235746788a460b32bc2e8ee471e53bf72856f6eb2f\": container with ID starting with bce99867a5f93185cc58d3235746788a460b32bc2e8ee471e53bf72856f6eb2f not found: ID does not exist" containerID="bce99867a5f93185cc58d3235746788a460b32bc2e8ee471e53bf72856f6eb2f" Oct 29 23:02:29.949140 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.949073 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce99867a5f93185cc58d3235746788a460b32bc2e8ee471e53bf72856f6eb2f"} err="failed to get container status \"bce99867a5f93185cc58d3235746788a460b32bc2e8ee471e53bf72856f6eb2f\": rpc error: code = NotFound desc = could not find container \"bce99867a5f93185cc58d3235746788a460b32bc2e8ee471e53bf72856f6eb2f\": container with ID starting with bce99867a5f93185cc58d3235746788a460b32bc2e8ee471e53bf72856f6eb2f not found: ID does not exist" Oct 29 23:02:29.949140 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.949092 4821 scope.go:122] "RemoveContainer" containerID="3d63bf0b80f83ed78e7cc83b12a8ca66f5b207ad0b5b44098d3d9531da1ed3bd" Oct 29 23:02:29.966700 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.966675 4821 scope.go:122] "RemoveContainer" containerID="3d63bf0b80f83ed78e7cc83b12a8ca66f5b207ad0b5b44098d3d9531da1ed3bd" Oct 29 23:02:29.967115 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:02:29.967088 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d63bf0b80f83ed78e7cc83b12a8ca66f5b207ad0b5b44098d3d9531da1ed3bd\": container with ID starting with 3d63bf0b80f83ed78e7cc83b12a8ca66f5b207ad0b5b44098d3d9531da1ed3bd not found: ID does not exist" containerID="3d63bf0b80f83ed78e7cc83b12a8ca66f5b207ad0b5b44098d3d9531da1ed3bd" Oct 29 23:02:29.967264 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.967119 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d63bf0b80f83ed78e7cc83b12a8ca66f5b207ad0b5b44098d3d9531da1ed3bd"} err="failed to get container status \"3d63bf0b80f83ed78e7cc83b12a8ca66f5b207ad0b5b44098d3d9531da1ed3bd\": rpc error: code = NotFound desc = could not find container \"3d63bf0b80f83ed78e7cc83b12a8ca66f5b207ad0b5b44098d3d9531da1ed3bd\": container with ID starting with 3d63bf0b80f83ed78e7cc83b12a8ca66f5b207ad0b5b44098d3d9531da1ed3bd not found: ID does not exist" Oct 29 23:02:29.990337 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.985604 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj"] Oct 29 23:02:29.990522 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:29.990433 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ck9pj"] Oct 29 23:02:30.030191 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.030152 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg"] Oct 29 23:02:30.034520 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.034488 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gl7kg"] Oct 29 23:02:30.124061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.124025 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq"] Oct 29 23:02:30.131396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.131361 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-gxghq"] Oct 29 23:02:30.171081 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.171049 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="240bea83-173e-4b4a-9989-9608028afac4" path="/var/lib/kubelet/pods/240bea83-173e-4b4a-9989-9608028afac4/volumes" Oct 29 23:02:30.171645 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.171629 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f99e4c4-51fa-4e77-b863-c8867c9a08c2" path="/var/lib/kubelet/pods/3f99e4c4-51fa-4e77-b863-c8867c9a08c2/volumes" Oct 29 23:02:30.172074 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.172061 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68d7791f-31f4-4552-91d0-120f640ad1fa" path="/var/lib/kubelet/pods/68d7791f-31f4-4552-91d0-120f640ad1fa/volumes" Oct 29 23:02:30.691004 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.690970 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-774bff-r77mm"] Oct 29 23:02:30.691980 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.691950 4821 cpu_manager.go:414] "containerMap: removing container" podUID="91606ba2-0df7-4092-8166-2e6c65bbfea2" containerName="revoke-keys" Oct 29 23:02:30.691980 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.691978 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91606ba2-0df7-4092-8166-2e6c65bbfea2" containerName="revoke-keys" Oct 29 23:02:30.692177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.691992 4821 cpu_manager.go:414] "containerMap: removing container" podUID="650729e2-7050-4990-b7f5-ec4fa4c51947" containerName="maas-api" Oct 29 23:02:30.692177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.692001 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="650729e2-7050-4990-b7f5-ec4fa4c51947" containerName="maas-api" Oct 29 23:02:30.692177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.692021 4821 cpu_manager.go:414] "containerMap: removing container" podUID="0bb6b077-1785-4673-9dc9-80258dcbecf3" containerName="istio-proxy" Oct 29 23:02:30.692177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.692029 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb6b077-1785-4673-9dc9-80258dcbecf3" containerName="istio-proxy" Oct 29 23:02:30.692177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.692044 4821 cpu_manager.go:414] "containerMap: removing container" podUID="240bea83-173e-4b4a-9989-9608028afac4" containerName="istio-proxy" Oct 29 23:02:30.692177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.692051 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="240bea83-173e-4b4a-9989-9608028afac4" containerName="istio-proxy" Oct 29 23:02:30.692177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.692072 4821 cpu_manager.go:414] "containerMap: removing container" podUID="03949775-cc5e-4c51-8179-f53230828245" containerName="maas-api" Oct 29 23:02:30.692177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.692080 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03949775-cc5e-4c51-8179-f53230828245" containerName="maas-api" Oct 29 23:02:30.692177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.692107 4821 cpu_manager.go:414] "containerMap: removing container" podUID="68d7791f-31f4-4552-91d0-120f640ad1fa" containerName="istio-proxy" Oct 29 23:02:30.692177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.692114 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d7791f-31f4-4552-91d0-120f640ad1fa" containerName="istio-proxy" Oct 29 23:02:30.692177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.692129 4821 cpu_manager.go:414] "containerMap: removing container" podUID="3f99e4c4-51fa-4e77-b863-c8867c9a08c2" containerName="istio-proxy" Oct 29 23:02:30.692177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.692136 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f99e4c4-51fa-4e77-b863-c8867c9a08c2" containerName="istio-proxy" Oct 29 23:02:30.692177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.692163 4821 cpu_manager.go:414] "containerMap: removing container" podUID="0c2e7b94-f0cf-424d-a296-c2fe8e60ff01" containerName="istio-proxy" Oct 29 23:02:30.692177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.692170 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2e7b94-f0cf-424d-a296-c2fe8e60ff01" containerName="istio-proxy" Oct 29 23:02:30.692877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.692187 4821 cpu_manager.go:414] "containerMap: removing container" podUID="6103a8bf-2a28-4991-8b55-fecb16515d7f" containerName="istio-proxy" Oct 29 23:02:30.692877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.692194 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6103a8bf-2a28-4991-8b55-fecb16515d7f" containerName="istio-proxy" Oct 29 23:02:30.692877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.692358 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb6b077-1785-4673-9dc9-80258dcbecf3" containerName="istio-proxy" Oct 29 23:02:30.692877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.692372 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="240bea83-173e-4b4a-9989-9608028afac4" containerName="istio-proxy" Oct 29 23:02:30.692877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.692386 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="03949775-cc5e-4c51-8179-f53230828245" containerName="maas-api" Oct 29 23:02:30.692877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.692400 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d7791f-31f4-4552-91d0-120f640ad1fa" containerName="istio-proxy" Oct 29 23:02:30.692877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.692411 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f99e4c4-51fa-4e77-b863-c8867c9a08c2" containerName="istio-proxy" Oct 29 23:02:30.692877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.692425 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2e7b94-f0cf-424d-a296-c2fe8e60ff01" containerName="istio-proxy" Oct 29 23:02:30.692877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.692434 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="6103a8bf-2a28-4991-8b55-fecb16515d7f" containerName="istio-proxy" Oct 29 23:02:30.692877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.692444 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="91606ba2-0df7-4092-8166-2e6c65bbfea2" containerName="revoke-keys" Oct 29 23:02:30.692877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.692454 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="650729e2-7050-4990-b7f5-ec4fa4c51947" containerName="maas-api" Oct 29 23:02:30.698296 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.698274 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-774bff-r77mm" Oct 29 23:02:30.710021 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.709991 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs"] Oct 29 23:02:30.714641 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.714615 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.723616 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.720558 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:02:30.723616 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.720613 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:02:30.723616 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.723080 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-774bff-r77mm"] Oct 29 23:02:30.745775 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.741101 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs"] Oct 29 23:02:30.807231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.798639 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vstzv\" (UniqueName: \"kubernetes.io/projected/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-kube-api-access-vstzv\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.807231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.798702 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-workload-socket\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.807231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.798773 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecaa3b06-01ff-4bf1-b709-47d221d32de0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-774bff-r77mm\" (UID: \"ecaa3b06-01ff-4bf1-b709-47d221d32de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-774bff-r77mm" Oct 29 23:02:30.807231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.798847 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-podinfo\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.807231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.798935 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istiod-ca-cert\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.807231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.798988 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecaa3b06-01ff-4bf1-b709-47d221d32de0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-774bff-r77mm\" (UID: \"ecaa3b06-01ff-4bf1-b709-47d221d32de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-774bff-r77mm" Oct 29 23:02:30.807231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.799078 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-token\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.807231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.799127 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-workload-certs\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.807231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.799220 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-data\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.807231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.799287 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-credential-socket\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.807231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.799310 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-envoy\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.807231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.799474 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsk7l\" (UniqueName: \"kubernetes.io/projected/ecaa3b06-01ff-4bf1-b709-47d221d32de0-kube-api-access-lsk7l\") pod \"maas-api-revoke-keys-e2e-api-b-774bff-r77mm\" (UID: \"ecaa3b06-01ff-4bf1-b709-47d221d32de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-774bff-r77mm" Oct 29 23:02:30.824226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.820351 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-53a768-openshift-default-dockercfg-b4hbz\"" Oct 29 23:02:30.908169 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.908135 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsk7l\" (UniqueName: \"kubernetes.io/projected/ecaa3b06-01ff-4bf1-b709-47d221d32de0-kube-api-access-lsk7l\") pod \"maas-api-revoke-keys-e2e-api-b-774bff-r77mm\" (UID: \"ecaa3b06-01ff-4bf1-b709-47d221d32de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-774bff-r77mm" Oct 29 23:02:30.908429 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.908187 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vstzv\" (UniqueName: \"kubernetes.io/projected/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-kube-api-access-vstzv\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.908429 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.908237 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-workload-socket\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.908429 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.908280 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecaa3b06-01ff-4bf1-b709-47d221d32de0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-774bff-r77mm\" (UID: \"ecaa3b06-01ff-4bf1-b709-47d221d32de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-774bff-r77mm" Oct 29 23:02:30.908429 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.908333 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-podinfo\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.908429 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.908390 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istiod-ca-cert\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.908429 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.908424 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecaa3b06-01ff-4bf1-b709-47d221d32de0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-774bff-r77mm\" (UID: \"ecaa3b06-01ff-4bf1-b709-47d221d32de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-774bff-r77mm" Oct 29 23:02:30.908758 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.908478 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-token\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.908758 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.908508 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-workload-certs\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.908758 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.908552 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-data\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.908758 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.908590 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-credential-socket\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.908758 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.908606 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-envoy\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.909484 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.909455 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-workload-socket\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.910174 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.910129 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecaa3b06-01ff-4bf1-b709-47d221d32de0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-774bff-r77mm\" (UID: \"ecaa3b06-01ff-4bf1-b709-47d221d32de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-774bff-r77mm" Oct 29 23:02:30.910611 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.910168 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-credential-socket\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.910611 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.910264 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istiod-ca-cert\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.910611 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.910533 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-data\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.910611 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.910577 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-workload-certs\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.912918 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.912893 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-envoy\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.913173 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.913153 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecaa3b06-01ff-4bf1-b709-47d221d32de0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-774bff-r77mm\" (UID: \"ecaa3b06-01ff-4bf1-b709-47d221d32de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-774bff-r77mm" Oct 29 23:02:30.914699 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.914672 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-podinfo\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.960900 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.960823 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-token\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.961124 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.961011 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vstzv\" (UniqueName: \"kubernetes.io/projected/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-kube-api-access-vstzv\") pod \"e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:30.974517 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:30.974184 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsk7l\" (UniqueName: \"kubernetes.io/projected/ecaa3b06-01ff-4bf1-b709-47d221d32de0-kube-api-access-lsk7l\") pod \"maas-api-revoke-keys-e2e-api-b-774bff-r77mm\" (UID: \"ecaa3b06-01ff-4bf1-b709-47d221d32de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-774bff-r77mm" Oct 29 23:02:31.013716 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:31.013680 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-774bff-r77mm" Oct 29 23:02:31.039695 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:31.039661 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:31.381260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:31.378818 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-774bff-r77mm"] Oct 29 23:02:31.400147 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:31.400114 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs"] Oct 29 23:02:31.405678 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:02:31.405641 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a4cc0f_6bd5_4098_87aa_ff42db1fbc10.slice/crio-b332ea071d9999517a8ccbeea21117a5b4be12a778cabc1a484f76516341c568 WatchSource:0}: Error finding container b332ea071d9999517a8ccbeea21117a5b4be12a778cabc1a484f76516341c568: Status 404 returned error can't find the container with id b332ea071d9999517a8ccbeea21117a5b4be12a778cabc1a484f76516341c568 Oct 29 23:02:31.409488 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:31.409455 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:02:31.409609 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:31.409540 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:02:31.409609 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:31.409588 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:02:31.611497 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:31.611456 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54"] Oct 29 23:02:31.612044 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:31.611997 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" podUID="14a9da2c-a970-4b40-a178-6f2215f8a44b" containerName="istio-proxy" containerID="cri-o://5109adf583dc9f6f52b28b238df4eee33f168cee2c52d6ae3779767388b37162" gracePeriod=30 Oct 29 23:02:32.005457 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.005326 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" event={"ID":"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10","Type":"ContainerStarted","Data":"fc92a35acaaa8e1ede0400b1dbc674112d76faec8ed5c23a2b0eb72e0bba1893"} Oct 29 23:02:32.006080 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.006029 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" event={"ID":"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10","Type":"ContainerStarted","Data":"b332ea071d9999517a8ccbeea21117a5b4be12a778cabc1a484f76516341c568"} Oct 29 23:02:32.009796 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.009752 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-774bff-r77mm" event={"ID":"ecaa3b06-01ff-4bf1-b709-47d221d32de0","Type":"ContainerStarted","Data":"9a7cc7639e7e018b362d6fd0d7c2eece496cb980dd1467a8d7d8fe98b062854a"} Oct 29 23:02:32.010020 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.010001 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-774bff-r77mm" event={"ID":"ecaa3b06-01ff-4bf1-b709-47d221d32de0","Type":"ContainerStarted","Data":"ed51f2f9f5d0c7332250b394fc9ff62301798846baed148b33408d75a1afe980"} Oct 29 23:02:32.040746 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.040713 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:32.042481 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.042451 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 29 23:02:32.042650 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.042521 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" podUID="83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 29 23:02:32.059671 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.059618 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" podStartSLOduration=2.05960329 podStartE2EDuration="2.05960329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:32.056549227 +0000 UTC m=+1205.997097137" watchObservedRunningTime="2026-10-29 23:02:32.05960329 +0000 UTC m=+1206.000151272" Oct 29 23:02:32.124110 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.123747 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-774bff-r77mm" podStartSLOduration=2.123727125 podStartE2EDuration="2.123727125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:32.109407539 +0000 UTC m=+1206.049955448" watchObservedRunningTime="2026-10-29 23:02:32.123727125 +0000 UTC m=+1206.064275046" Oct 29 23:02:32.668665 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.668630 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq"] Oct 29 23:02:32.674111 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.674088 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.694707 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.693679 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-1e1c46-openshift-default-dockercfg-r5fs5\"" Oct 29 23:02:32.695419 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.695152 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq"] Oct 29 23:02:32.769885 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.769576 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-data\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.769885 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.769672 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-podinfo\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.769885 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.769700 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-envoy\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.769885 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.769739 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-workload-socket\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.769885 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.769832 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e08c58b-702a-437f-bb87-1a4bc4727f78-istiod-ca-cert\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.769885 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.769865 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-token\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.770377 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.769902 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-credential-socket\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.770377 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.769990 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-workload-certs\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.770377 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.770017 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ndwr\" (UniqueName: \"kubernetes.io/projected/9e08c58b-702a-437f-bb87-1a4bc4727f78-kube-api-access-9ndwr\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.871375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.871311 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-data\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.871586 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.871400 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-podinfo\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.871586 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.871420 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-envoy\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.871586 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.871454 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-workload-socket\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.871586 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.871534 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e08c58b-702a-437f-bb87-1a4bc4727f78-istiod-ca-cert\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.871586 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.871564 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-token\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.871586 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.871590 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-credential-socket\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.871923 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.871657 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-workload-certs\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.871923 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.871674 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ndwr\" (UniqueName: \"kubernetes.io/projected/9e08c58b-702a-437f-bb87-1a4bc4727f78-kube-api-access-9ndwr\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.872721 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.872622 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-data\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.873437 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.873417 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e08c58b-702a-437f-bb87-1a4bc4727f78-istiod-ca-cert\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.873932 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.873909 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-credential-socket\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.874295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.874270 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-workload-certs\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.874559 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.874538 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-workload-socket\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.877156 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.877129 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-podinfo\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.883554 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.883528 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-envoy\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.898247 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.898194 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ndwr\" (UniqueName: \"kubernetes.io/projected/9e08c58b-702a-437f-bb87-1a4bc4727f78-kube-api-access-9ndwr\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.901592 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.901565 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-token\") pod \"e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:32.998964 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:32.997521 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:33.024142 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:33.024108 4821 generic.go:356] "Generic (PLEG): container finished" podID="ecaa3b06-01ff-4bf1-b709-47d221d32de0" containerID="9a7cc7639e7e018b362d6fd0d7c2eece496cb980dd1467a8d7d8fe98b062854a" exitCode=0 Oct 29 23:02:33.024592 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:33.024190 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-774bff-r77mm" event={"ID":"ecaa3b06-01ff-4bf1-b709-47d221d32de0","Type":"ContainerDied","Data":"9a7cc7639e7e018b362d6fd0d7c2eece496cb980dd1467a8d7d8fe98b062854a"} Oct 29 23:02:33.049652 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:33.049578 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:33.367276 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:33.367245 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq"] Oct 29 23:02:33.368629 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:02:33.368591 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e08c58b_702a_437f_bb87_1a4bc4727f78.slice/crio-e79b2218fb5fc6fae2babbceed8cb39731f7a3d1e1c05fa85c4ef8076a9a95eb WatchSource:0}: Error finding container e79b2218fb5fc6fae2babbceed8cb39731f7a3d1e1c05fa85c4ef8076a9a95eb: Status 404 returned error can't find the container with id e79b2218fb5fc6fae2babbceed8cb39731f7a3d1e1c05fa85c4ef8076a9a95eb Oct 29 23:02:33.373665 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:33.373620 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:02:33.373781 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:33.373718 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:02:33.373781 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:33.373766 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:02:33.800833 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:33.800739 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5"] Oct 29 23:02:33.805957 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:33.805888 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:33.835191 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:33.835158 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5"] Oct 29 23:02:33.853334 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:33.853166 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-d67deb-openshift-default-dockercfg-x249h\"" Oct 29 23:02:33.901009 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:33.900965 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-762xw\" (UniqueName: \"kubernetes.io/projected/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-kube-api-access-762xw\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:33.901212 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:33.901038 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-envoy\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:33.901276 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:33.901221 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-token\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:33.901367 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:33.901348 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-workload-certs\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:33.901419 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:33.901387 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-podinfo\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:33.901616 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:33.901596 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-data\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:33.901661 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:33.901634 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istiod-ca-cert\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:33.901718 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:33.901703 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-credential-socket\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:33.901817 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:33.901801 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-workload-socket\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:34.007602 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.007563 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-data\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:34.008506 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.008483 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istiod-ca-cert\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:34.009593 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.009547 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-credential-socket\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:34.010384 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.010150 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-credential-socket\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:34.010580 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.008428 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-data\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:34.010710 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.010246 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istiod-ca-cert\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:34.011291 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.010941 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-workload-socket\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:34.011534 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.011514 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-762xw\" (UniqueName: \"kubernetes.io/projected/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-kube-api-access-762xw\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:34.012250 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.012228 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-envoy\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:34.012933 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.012912 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-token\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:34.013454 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.013417 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-workload-certs\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:34.013895 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.013879 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-podinfo\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:34.014558 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.011235 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-workload-socket\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:34.014692 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.013815 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-workload-certs\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:34.081280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.038038 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-envoy\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:34.081280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.057122 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-podinfo\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:34.082892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.082789 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-762xw\" (UniqueName: \"kubernetes.io/projected/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-kube-api-access-762xw\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:34.092807 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.092720 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-token\") pod \"e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:34.102626 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.101371 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" event={"ID":"9e08c58b-702a-437f-bb87-1a4bc4727f78","Type":"ContainerStarted","Data":"159bb96543d6f0d3c192fa136d44fbc179e9692be2614277723c82383d25b5fa"} Oct 29 23:02:34.102626 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.101436 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" event={"ID":"9e08c58b-702a-437f-bb87-1a4bc4727f78","Type":"ContainerStarted","Data":"e79b2218fb5fc6fae2babbceed8cb39731f7a3d1e1c05fa85c4ef8076a9a95eb"} Oct 29 23:02:34.102626 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.102544 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:34.126222 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.124507 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:02:34.126222 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.126007 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:34.171372 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.169686 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" podStartSLOduration=2.169669527 podStartE2EDuration="2.169669527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:34.165586408 +0000 UTC m=+1208.106134318" watchObservedRunningTime="2026-10-29 23:02:34.169669527 +0000 UTC m=+1208.110217437" Oct 29 23:02:34.460895 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.460859 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-774bff-r77mm" Oct 29 23:02:34.548268 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.540699 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecaa3b06-01ff-4bf1-b709-47d221d32de0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecaa3b06-01ff-4bf1-b709-47d221d32de0-maas-api-xks-ca\") pod \"ecaa3b06-01ff-4bf1-b709-47d221d32de0\" (UID: \"ecaa3b06-01ff-4bf1-b709-47d221d32de0\") " Oct 29 23:02:34.548268 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.540855 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecaa3b06-01ff-4bf1-b709-47d221d32de0-kube-api-access-lsk7l\" (UniqueName: \"kubernetes.io/projected/ecaa3b06-01ff-4bf1-b709-47d221d32de0-kube-api-access-lsk7l\") pod \"ecaa3b06-01ff-4bf1-b709-47d221d32de0\" (UID: \"ecaa3b06-01ff-4bf1-b709-47d221d32de0\") " Oct 29 23:02:34.548268 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.541106 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecaa3b06-01ff-4bf1-b709-47d221d32de0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecaa3b06-01ff-4bf1-b709-47d221d32de0-maas-api-openshift-ca\") pod \"ecaa3b06-01ff-4bf1-b709-47d221d32de0\" (UID: \"ecaa3b06-01ff-4bf1-b709-47d221d32de0\") " Oct 29 23:02:34.548268 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.544426 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecaa3b06-01ff-4bf1-b709-47d221d32de0-maas-api-openshift-ca" pod "ecaa3b06-01ff-4bf1-b709-47d221d32de0" (UID: "ecaa3b06-01ff-4bf1-b709-47d221d32de0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:34.548268 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.547151 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecaa3b06-01ff-4bf1-b709-47d221d32de0-maas-api-xks-ca" pod "ecaa3b06-01ff-4bf1-b709-47d221d32de0" (UID: "ecaa3b06-01ff-4bf1-b709-47d221d32de0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:02:34.549564 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.549532 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecaa3b06-01ff-4bf1-b709-47d221d32de0-kube-api-access-lsk7l" pod "ecaa3b06-01ff-4bf1-b709-47d221d32de0" (UID: "ecaa3b06-01ff-4bf1-b709-47d221d32de0"). InnerVolumeSpecName "kube-api-access-lsk7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:34.640978 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.640958 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5"] Oct 29 23:02:34.644173 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.644144 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsk7l\" (UniqueName: \"kubernetes.io/projected/ecaa3b06-01ff-4bf1-b709-47d221d32de0-kube-api-access-lsk7l\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:34.644173 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.644171 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecaa3b06-01ff-4bf1-b709-47d221d32de0-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:34.644386 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.644183 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecaa3b06-01ff-4bf1-b709-47d221d32de0-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:34.644386 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:02:34.644178 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e845eec_edd8_4ce2_ad4c_fbfc15fead43.slice/crio-57ee585fd21f061fec1ea4eae02f6f1425ee6df139aba84d242960857ba0c39c WatchSource:0}: Error finding container 57ee585fd21f061fec1ea4eae02f6f1425ee6df139aba84d242960857ba0c39c: Status 404 returned error can't find the container with id 57ee585fd21f061fec1ea4eae02f6f1425ee6df139aba84d242960857ba0c39c Oct 29 23:02:34.647185 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.647136 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:02:34.647310 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.647244 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:02:34.647310 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.647291 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:02:34.998154 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:34.998063 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:35.005429 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:35.005398 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:35.112614 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:35.112584 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-774bff-r77mm" Oct 29 23:02:35.112614 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:35.112591 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-774bff-r77mm" event={"ID":"ecaa3b06-01ff-4bf1-b709-47d221d32de0","Type":"ContainerDied","Data":"ed51f2f9f5d0c7332250b394fc9ff62301798846baed148b33408d75a1afe980"} Oct 29 23:02:35.113158 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:35.112648 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed51f2f9f5d0c7332250b394fc9ff62301798846baed148b33408d75a1afe980" Oct 29 23:02:35.114750 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:35.114719 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" event={"ID":"1e845eec-edd8-4ce2-ad4c-fbfc15fead43","Type":"ContainerStarted","Data":"0a7dc024e82fada74a07789f502359da954a9573b26b37e5715885b2ee9f6fd4"} Oct 29 23:02:35.114876 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:35.114774 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" event={"ID":"1e845eec-edd8-4ce2-ad4c-fbfc15fead43","Type":"ContainerStarted","Data":"57ee585fd21f061fec1ea4eae02f6f1425ee6df139aba84d242960857ba0c39c"} Oct 29 23:02:35.115555 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:35.115454 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:35.116270 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:35.116252 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:02:35.127242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:35.127221 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:35.133701 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:35.133682 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:35.164384 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:35.164330 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" podStartSLOduration=2.1643107600000002 podStartE2EDuration="2.16431076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:35.161604751 +0000 UTC m=+1209.102152663" watchObservedRunningTime="2026-10-29 23:02:35.16431076 +0000 UTC m=+1209.104858671" Oct 29 23:02:35.467778 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:35.467739 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:02:35.468025 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:35.467806 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" podUID="14a9da2c-a970-4b40-a178-6f2215f8a44b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:02:36.147695 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:36.147497 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:36.149181 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:36.149162 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:02:36.932706 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:36.932674 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:02:37.051438 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.051388 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-data\") pod \"14a9da2c-a970-4b40-a178-6f2215f8a44b\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " Oct 29 23:02:37.051622 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.051445 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-token\" (UniqueName: \"kubernetes.io/projected/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-token\") pod \"14a9da2c-a970-4b40-a178-6f2215f8a44b\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " Oct 29 23:02:37.051622 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.051489 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-workload-certs\") pod \"14a9da2c-a970-4b40-a178-6f2215f8a44b\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " Oct 29 23:02:37.051763 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.051630 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14a9da2c-a970-4b40-a178-6f2215f8a44b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a9da2c-a970-4b40-a178-6f2215f8a44b-istiod-ca-cert\") pod \"14a9da2c-a970-4b40-a178-6f2215f8a44b\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " Oct 29 23:02:37.051763 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.051687 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a9da2c-a970-4b40-a178-6f2215f8a44b-kube-api-access-bmdl6\" (UniqueName: \"kubernetes.io/projected/14a9da2c-a970-4b40-a178-6f2215f8a44b-kube-api-access-bmdl6\") pod \"14a9da2c-a970-4b40-a178-6f2215f8a44b\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " Oct 29 23:02:37.051763 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.051737 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-podinfo\") pod \"14a9da2c-a970-4b40-a178-6f2215f8a44b\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " Oct 29 23:02:37.051923 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.051801 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-credential-socket\") pod \"14a9da2c-a970-4b40-a178-6f2215f8a44b\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " Oct 29 23:02:37.051923 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.051864 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-workload-socket\") pod \"14a9da2c-a970-4b40-a178-6f2215f8a44b\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " Oct 29 23:02:37.051923 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.051893 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-envoy\") pod \"14a9da2c-a970-4b40-a178-6f2215f8a44b\" (UID: \"14a9da2c-a970-4b40-a178-6f2215f8a44b\") " Oct 29 23:02:37.052491 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.052372 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a9da2c-a970-4b40-a178-6f2215f8a44b-istiod-ca-cert" pod "14a9da2c-a970-4b40-a178-6f2215f8a44b" (UID: "14a9da2c-a970-4b40-a178-6f2215f8a44b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:37.053732 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.053099 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-workload-certs" pod "14a9da2c-a970-4b40-a178-6f2215f8a44b" (UID: "14a9da2c-a970-4b40-a178-6f2215f8a44b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:37.053732 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.053312 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-data" pod "14a9da2c-a970-4b40-a178-6f2215f8a44b" (UID: "14a9da2c-a970-4b40-a178-6f2215f8a44b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:37.053732 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.053503 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-workload-socket" pod "14a9da2c-a970-4b40-a178-6f2215f8a44b" (UID: "14a9da2c-a970-4b40-a178-6f2215f8a44b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:37.053732 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.053647 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a9da2c-a970-4b40-a178-6f2215f8a44b-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.053732 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.053671 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.053732 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.053685 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.053732 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.053699 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.054474 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.054449 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-credential-socket" pod "14a9da2c-a970-4b40-a178-6f2215f8a44b" (UID: "14a9da2c-a970-4b40-a178-6f2215f8a44b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:37.055693 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.055665 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-envoy" pod "14a9da2c-a970-4b40-a178-6f2215f8a44b" (UID: "14a9da2c-a970-4b40-a178-6f2215f8a44b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:37.058630 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.058438 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-token" pod "14a9da2c-a970-4b40-a178-6f2215f8a44b" (UID: "14a9da2c-a970-4b40-a178-6f2215f8a44b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:37.060609 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.060563 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-podinfo" pod "14a9da2c-a970-4b40-a178-6f2215f8a44b" (UID: "14a9da2c-a970-4b40-a178-6f2215f8a44b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:02:37.060711 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.060638 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a9da2c-a970-4b40-a178-6f2215f8a44b-kube-api-access-bmdl6" pod "14a9da2c-a970-4b40-a178-6f2215f8a44b" (UID: "14a9da2c-a970-4b40-a178-6f2215f8a44b"). InnerVolumeSpecName "kube-api-access-bmdl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:37.155561 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.155517 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmdl6\" (UniqueName: \"kubernetes.io/projected/14a9da2c-a970-4b40-a178-6f2215f8a44b-kube-api-access-bmdl6\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.155561 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.155547 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.155561 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.155556 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.155561 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.155565 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.155561 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.155573 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a9da2c-a970-4b40-a178-6f2215f8a44b-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:37.157908 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.157879 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" event={"ID":"14a9da2c-a970-4b40-a178-6f2215f8a44b","Type":"ContainerDied","Data":"5109adf583dc9f6f52b28b238df4eee33f168cee2c52d6ae3779767388b37162"} Oct 29 23:02:37.161273 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.158336 4821 scope.go:122] "RemoveContainer" containerID="5109adf583dc9f6f52b28b238df4eee33f168cee2c52d6ae3779767388b37162" Oct 29 23:02:37.161273 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.158363 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" Oct 29 23:02:37.161273 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.157845 4821 generic.go:356] "Generic (PLEG): container finished" podID="14a9da2c-a970-4b40-a178-6f2215f8a44b" containerID="5109adf583dc9f6f52b28b238df4eee33f168cee2c52d6ae3779767388b37162" exitCode=0 Oct 29 23:02:37.161273 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.158592 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54" event={"ID":"14a9da2c-a970-4b40-a178-6f2215f8a44b","Type":"ContainerDied","Data":"2565bdc0afd272e341feddd9b2ccc2fc521f42319e4a4409eddbf2ea410de34c"} Oct 29 23:02:37.184399 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.184376 4821 scope.go:122] "RemoveContainer" containerID="5109adf583dc9f6f52b28b238df4eee33f168cee2c52d6ae3779767388b37162" Oct 29 23:02:37.184750 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:02:37.184719 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5109adf583dc9f6f52b28b238df4eee33f168cee2c52d6ae3779767388b37162\": container with ID starting with 5109adf583dc9f6f52b28b238df4eee33f168cee2c52d6ae3779767388b37162 not found: ID does not exist" containerID="5109adf583dc9f6f52b28b238df4eee33f168cee2c52d6ae3779767388b37162" Oct 29 23:02:37.184874 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.184762 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5109adf583dc9f6f52b28b238df4eee33f168cee2c52d6ae3779767388b37162"} err="failed to get container status \"5109adf583dc9f6f52b28b238df4eee33f168cee2c52d6ae3779767388b37162\": rpc error: code = NotFound desc = could not find container \"5109adf583dc9f6f52b28b238df4eee33f168cee2c52d6ae3779767388b37162\": container with ID starting with 5109adf583dc9f6f52b28b238df4eee33f168cee2c52d6ae3779767388b37162 not found: ID does not exist" Oct 29 23:02:37.226288 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.226256 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54"] Oct 29 23:02:37.228067 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.228035 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-13aa41-openshift-default-59db78b859-c6c54"] Oct 29 23:02:37.370948 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.370914 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g"] Oct 29 23:02:37.371597 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.371576 4821 cpu_manager.go:414] "containerMap: removing container" podUID="14a9da2c-a970-4b40-a178-6f2215f8a44b" containerName="istio-proxy" Oct 29 23:02:37.371730 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.371597 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a9da2c-a970-4b40-a178-6f2215f8a44b" containerName="istio-proxy" Oct 29 23:02:37.371730 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.371631 4821 cpu_manager.go:414] "containerMap: removing container" podUID="ecaa3b06-01ff-4bf1-b709-47d221d32de0" containerName="revoke-keys" Oct 29 23:02:37.371730 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.371638 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecaa3b06-01ff-4bf1-b709-47d221d32de0" containerName="revoke-keys" Oct 29 23:02:37.371905 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.371816 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecaa3b06-01ff-4bf1-b709-47d221d32de0" containerName="revoke-keys" Oct 29 23:02:37.371905 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.371841 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a9da2c-a970-4b40-a178-6f2215f8a44b" containerName="istio-proxy" Oct 29 23:02:37.376842 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.376819 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g" Oct 29 23:02:37.393328 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.393298 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-53a768\"" Oct 29 23:02:37.396505 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.396478 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-53a768-dockercfg-6dstb\"" Oct 29 23:02:37.397591 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.397567 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g"] Oct 29 23:02:37.429404 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.426460 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn"] Oct 29 23:02:37.460224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.458527 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn" Oct 29 23:02:37.511301 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.460532 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a84143c5-852c-4ebb-8aee-c5e3dd323ef6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g\" (UID: \"a84143c5-852c-4ebb-8aee-c5e3dd323ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g" Oct 29 23:02:37.511301 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.460682 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtssv\" (UniqueName: \"kubernetes.io/projected/a84143c5-852c-4ebb-8aee-c5e3dd323ef6-kube-api-access-gtssv\") pod \"payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g\" (UID: \"a84143c5-852c-4ebb-8aee-c5e3dd323ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g" Oct 29 23:02:37.572296 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.561677 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs5jz\" (UniqueName: \"kubernetes.io/projected/1425a1f4-640b-4c45-bae9-7e0d4b66afcf-kube-api-access-cs5jz\") pod \"payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn\" (UID: \"1425a1f4-640b-4c45-bae9-7e0d4b66afcf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn" Oct 29 23:02:37.572296 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.561789 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a84143c5-852c-4ebb-8aee-c5e3dd323ef6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g\" (UID: \"a84143c5-852c-4ebb-8aee-c5e3dd323ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g" Oct 29 23:02:37.572296 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.561872 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtssv\" (UniqueName: \"kubernetes.io/projected/a84143c5-852c-4ebb-8aee-c5e3dd323ef6-kube-api-access-gtssv\") pod \"payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g\" (UID: \"a84143c5-852c-4ebb-8aee-c5e3dd323ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g" Oct 29 23:02:37.572296 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.562000 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1425a1f4-640b-4c45-bae9-7e0d4b66afcf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn\" (UID: \"1425a1f4-640b-4c45-bae9-7e0d4b66afcf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn" Oct 29 23:02:37.572296 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.563257 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a84143c5-852c-4ebb-8aee-c5e3dd323ef6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g\" (UID: \"a84143c5-852c-4ebb-8aee-c5e3dd323ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g" Oct 29 23:02:37.657720 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.657680 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn"] Oct 29 23:02:37.681272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.679164 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1425a1f4-640b-4c45-bae9-7e0d4b66afcf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn\" (UID: \"1425a1f4-640b-4c45-bae9-7e0d4b66afcf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn" Oct 29 23:02:37.681272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.679677 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs5jz\" (UniqueName: \"kubernetes.io/projected/1425a1f4-640b-4c45-bae9-7e0d4b66afcf-kube-api-access-cs5jz\") pod \"payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn\" (UID: \"1425a1f4-640b-4c45-bae9-7e0d4b66afcf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn" Oct 29 23:02:37.681272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.680735 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1425a1f4-640b-4c45-bae9-7e0d4b66afcf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn\" (UID: \"1425a1f4-640b-4c45-bae9-7e0d4b66afcf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn" Oct 29 23:02:37.711336 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.702274 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtssv\" (UniqueName: \"kubernetes.io/projected/a84143c5-852c-4ebb-8aee-c5e3dd323ef6-kube-api-access-gtssv\") pod \"payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g\" (UID: \"a84143c5-852c-4ebb-8aee-c5e3dd323ef6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g" Oct 29 23:02:37.756234 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.753021 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs5jz\" (UniqueName: \"kubernetes.io/projected/1425a1f4-640b-4c45-bae9-7e0d4b66afcf-kube-api-access-cs5jz\") pod \"payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn\" (UID: \"1425a1f4-640b-4c45-bae9-7e0d4b66afcf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn" Oct 29 23:02:38.001306 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.996028 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn" Oct 29 23:02:38.001306 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:37.998674 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g" Oct 29 23:02:38.185702 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:38.185668 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14a9da2c-a970-4b40-a178-6f2215f8a44b" path="/var/lib/kubelet/pods/14a9da2c-a970-4b40-a178-6f2215f8a44b/volumes" Oct 29 23:02:38.363440 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:38.361033 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn"] Oct 29 23:02:38.363560 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:02:38.361266 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1425a1f4_640b_4c45_bae9_7e0d4b66afcf.slice/crio-54000fab6f816e127de11d87193a11a9f9a3ef3d53eaa52681f73a32753d3a75 WatchSource:0}: Error finding container 54000fab6f816e127de11d87193a11a9f9a3ef3d53eaa52681f73a32753d3a75: Status 404 returned error can't find the container with id 54000fab6f816e127de11d87193a11a9f9a3ef3d53eaa52681f73a32753d3a75 Oct 29 23:02:38.367382 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:38.367318 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g"] Oct 29 23:02:39.188153 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:39.188114 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g" event={"ID":"a84143c5-852c-4ebb-8aee-c5e3dd323ef6","Type":"ContainerStarted","Data":"dde36c2a524b2e1da1ec2a6b280d5d00dec65e1bcc0adde2d9c159677a8df955"} Oct 29 23:02:39.188659 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:39.188182 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g" Oct 29 23:02:39.188659 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:39.188224 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g" event={"ID":"a84143c5-852c-4ebb-8aee-c5e3dd323ef6","Type":"ContainerStarted","Data":"08dbb5024f9d427a54dcbf77c44be5c84bd32e351e9023dfdaae17ba7d115407"} Oct 29 23:02:39.190101 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:39.190074 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn" event={"ID":"1425a1f4-640b-4c45-bae9-7e0d4b66afcf","Type":"ContainerStarted","Data":"a5133bbaeea0e3e64a6bf89d5f86db87684fc323dd3d4ca9c0c0544a27aa007a"} Oct 29 23:02:39.190232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:39.190129 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn" event={"ID":"1425a1f4-640b-4c45-bae9-7e0d4b66afcf","Type":"ContainerStarted","Data":"54000fab6f816e127de11d87193a11a9f9a3ef3d53eaa52681f73a32753d3a75"} Oct 29 23:02:39.190232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:39.190158 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn" Oct 29 23:02:39.236483 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:39.236429 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g" podStartSLOduration=2.236412608 podStartE2EDuration="2.236412608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:39.236327406 +0000 UTC m=+1213.176875316" watchObservedRunningTime="2026-10-29 23:02:39.236412608 +0000 UTC m=+1213.176960518" Oct 29 23:02:39.287891 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:39.287825 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn" podStartSLOduration=2.28780589 podStartE2EDuration="2.28780589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:39.282669235 +0000 UTC m=+1213.223217146" watchObservedRunningTime="2026-10-29 23:02:39.28780589 +0000 UTC m=+1213.228353802" Oct 29 23:02:40.185341 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:40.185308 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz"] Oct 29 23:02:40.189081 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:40.189058 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz" Oct 29 23:02:40.206963 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:40.206931 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-1e1c46\"" Oct 29 23:02:40.220451 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:40.220425 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-1e1c46-dockercfg-dd4t2\"" Oct 29 23:02:40.237948 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:40.227506 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz"] Oct 29 23:02:40.285400 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:40.276259 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64bqp\" (UniqueName: \"kubernetes.io/projected/8a5244fa-2331-48c6-b32f-3369cad8bd3c-kube-api-access-64bqp\") pod \"payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz\" (UID: \"8a5244fa-2331-48c6-b32f-3369cad8bd3c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz" Oct 29 23:02:40.285400 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:40.276989 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5244fa-2331-48c6-b32f-3369cad8bd3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz\" (UID: \"8a5244fa-2331-48c6-b32f-3369cad8bd3c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz" Oct 29 23:02:40.419522 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:40.419479 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp"] Oct 29 23:02:40.431354 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:40.424650 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64bqp\" (UniqueName: \"kubernetes.io/projected/8a5244fa-2331-48c6-b32f-3369cad8bd3c-kube-api-access-64bqp\") pod \"payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz\" (UID: \"8a5244fa-2331-48c6-b32f-3369cad8bd3c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz" Oct 29 23:02:40.431354 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:40.424851 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5244fa-2331-48c6-b32f-3369cad8bd3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz\" (UID: \"8a5244fa-2331-48c6-b32f-3369cad8bd3c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz" Oct 29 23:02:40.431354 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:40.425801 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5244fa-2331-48c6-b32f-3369cad8bd3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz\" (UID: \"8a5244fa-2331-48c6-b32f-3369cad8bd3c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz" Oct 29 23:02:40.437154 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:40.434921 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp"] Oct 29 23:02:40.437154 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:40.435040 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp" Oct 29 23:02:40.497225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:40.495158 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64bqp\" (UniqueName: \"kubernetes.io/projected/8a5244fa-2331-48c6-b32f-3369cad8bd3c-kube-api-access-64bqp\") pod \"payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz\" (UID: \"8a5244fa-2331-48c6-b32f-3369cad8bd3c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz" Oct 29 23:02:40.514277 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:40.511186 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz" Oct 29 23:02:40.631228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:40.629616 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b740bdc5-90ed-42bd-a889-3582c45c8062-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp\" (UID: \"b740bdc5-90ed-42bd-a889-3582c45c8062\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp" Oct 29 23:02:40.631228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:40.629794 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h2lm\" (UniqueName: \"kubernetes.io/projected/b740bdc5-90ed-42bd-a889-3582c45c8062-kube-api-access-6h2lm\") pod \"payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp\" (UID: \"b740bdc5-90ed-42bd-a889-3582c45c8062\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp" Oct 29 23:02:40.731356 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:40.731257 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b740bdc5-90ed-42bd-a889-3582c45c8062-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp\" (UID: \"b740bdc5-90ed-42bd-a889-3582c45c8062\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp" Oct 29 23:02:40.731523 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:40.731459 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h2lm\" (UniqueName: \"kubernetes.io/projected/b740bdc5-90ed-42bd-a889-3582c45c8062-kube-api-access-6h2lm\") pod \"payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp\" (UID: \"b740bdc5-90ed-42bd-a889-3582c45c8062\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp" Oct 29 23:02:40.731964 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:40.731941 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b740bdc5-90ed-42bd-a889-3582c45c8062-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp\" (UID: \"b740bdc5-90ed-42bd-a889-3582c45c8062\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp" Oct 29 23:02:40.758889 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:40.758857 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h2lm\" (UniqueName: \"kubernetes.io/projected/b740bdc5-90ed-42bd-a889-3582c45c8062-kube-api-access-6h2lm\") pod \"payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp\" (UID: \"b740bdc5-90ed-42bd-a889-3582c45c8062\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp" Oct 29 23:02:40.785851 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:40.785818 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp" Oct 29 23:02:40.881329 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:40.879317 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz"] Oct 29 23:02:40.884342 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:02:40.884295 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5244fa_2331_48c6_b32f_3369cad8bd3c.slice/crio-cf667644006e3e00f7277da6f03aa37d05e94676eb550d24e307cafe78f37b55 WatchSource:0}: Error finding container cf667644006e3e00f7277da6f03aa37d05e94676eb550d24e307cafe78f37b55: Status 404 returned error can't find the container with id cf667644006e3e00f7277da6f03aa37d05e94676eb550d24e307cafe78f37b55 Oct 29 23:02:41.101475 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.101442 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp"] Oct 29 23:02:41.102450 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:02:41.102422 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb740bdc5_90ed_42bd_a889_3582c45c8062.slice/crio-9b10d4a27d4360523d8a72f693c6894ec50cb6bd2e740f97ffe25b92697685e7 WatchSource:0}: Error finding container 9b10d4a27d4360523d8a72f693c6894ec50cb6bd2e740f97ffe25b92697685e7: Status 404 returned error can't find the container with id 9b10d4a27d4360523d8a72f693c6894ec50cb6bd2e740f97ffe25b92697685e7 Oct 29 23:02:41.210157 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.210118 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz" event={"ID":"8a5244fa-2331-48c6-b32f-3369cad8bd3c","Type":"ContainerStarted","Data":"a95e7b6e4bb759dd09bddb50f53906c220bdecbb1c2af1d96e8f25b7e1359346"} Oct 29 23:02:41.210615 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.210185 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz" event={"ID":"8a5244fa-2331-48c6-b32f-3369cad8bd3c","Type":"ContainerStarted","Data":"cf667644006e3e00f7277da6f03aa37d05e94676eb550d24e307cafe78f37b55"} Oct 29 23:02:41.210615 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.210260 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz" Oct 29 23:02:41.212306 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.212270 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp" event={"ID":"b740bdc5-90ed-42bd-a889-3582c45c8062","Type":"ContainerStarted","Data":"f8d660957e40a9690ac49ba841b40b99d81753919dae8650ab9e6df9dd702481"} Oct 29 23:02:41.212465 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.212334 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp" event={"ID":"b740bdc5-90ed-42bd-a889-3582c45c8062","Type":"ContainerStarted","Data":"9b10d4a27d4360523d8a72f693c6894ec50cb6bd2e740f97ffe25b92697685e7"} Oct 29 23:02:41.212465 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.212362 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp" Oct 29 23:02:41.253642 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.253583 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz" podStartSLOduration=1.253568172 podStartE2EDuration="1.253568172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:41.252332552 +0000 UTC m=+1215.192880462" watchObservedRunningTime="2026-10-29 23:02:41.253568172 +0000 UTC m=+1215.194116082" Oct 29 23:02:41.288741 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.288657 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5"] Oct 29 23:02:41.294002 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.293962 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.317406 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.316766 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp" podStartSLOduration=1.316748381 podStartE2EDuration="1.316748381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:41.314112554 +0000 UTC m=+1215.254660466" watchObservedRunningTime="2026-10-29 23:02:41.316748381 +0000 UTC m=+1215.257296294" Oct 29 23:02:41.318568 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.318541 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-0266dd-openshift-default-dockercfg-gr5jd\"" Oct 29 23:02:41.461245 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.460870 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8ls9\" (UniqueName: \"kubernetes.io/projected/e778eb53-b230-451e-b7ae-3d8512366c59-kube-api-access-q8ls9\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.461245 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.460992 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e778eb53-b230-451e-b7ae-3d8512366c59-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.461245 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.461034 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.461245 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.461104 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.461245 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.461146 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.461245 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.461180 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e778eb53-b230-451e-b7ae-3d8512366c59-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.461724 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.461468 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.461724 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.461607 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e778eb53-b230-451e-b7ae-3d8512366c59-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.461724 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.461710 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.477296 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.463689 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5"] Oct 29 23:02:41.551567 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.551457 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-774bff-b8475597b-gkstk"] Oct 29 23:02:41.563675 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.563634 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8ls9\" (UniqueName: \"kubernetes.io/projected/e778eb53-b230-451e-b7ae-3d8512366c59-kube-api-access-q8ls9\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.563857 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.563720 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e778eb53-b230-451e-b7ae-3d8512366c59-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.563857 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.563739 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.563857 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.563781 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.563857 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.563809 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.563857 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.563835 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e778eb53-b230-451e-b7ae-3d8512366c59-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.564238 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.563900 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.564238 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.563976 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e778eb53-b230-451e-b7ae-3d8512366c59-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.564238 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.564059 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.564400 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.564238 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.564400 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.564369 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.564605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.564577 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.564746 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.564605 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.565039 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.565014 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e778eb53-b230-451e-b7ae-3d8512366c59-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.566994 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.566963 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.568961 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.568933 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e778eb53-b230-451e-b7ae-3d8512366c59-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.590923 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.590881 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8ls9\" (UniqueName: \"kubernetes.io/projected/e778eb53-b230-451e-b7ae-3d8512366c59-kube-api-access-q8ls9\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.592302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.592273 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e778eb53-b230-451e-b7ae-3d8512366c59-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:41.709613 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:41.709571 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:42.086558 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:42.086533 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5"] Oct 29 23:02:42.087188 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:02:42.087153 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode778eb53_b230_451e_b7ae_3d8512366c59.slice/crio-c7256a75f7f86cccc9d8a466bc0ccfe17cae2269e9afbdb37ff7bdeed8f741fb WatchSource:0}: Error finding container c7256a75f7f86cccc9d8a466bc0ccfe17cae2269e9afbdb37ff7bdeed8f741fb: Status 404 returned error can't find the container with id c7256a75f7f86cccc9d8a466bc0ccfe17cae2269e9afbdb37ff7bdeed8f741fb Oct 29 23:02:42.090145 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:42.090110 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:02:42.090275 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:42.090188 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:02:42.090353 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:42.090287 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:02:42.224919 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:42.224883 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" event={"ID":"e778eb53-b230-451e-b7ae-3d8512366c59","Type":"ContainerStarted","Data":"6720048b4ee28892faf8bbec09bddd8c5caa536c4959d46171a078a6908fac02"} Oct 29 23:02:42.225423 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:42.224953 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" event={"ID":"e778eb53-b230-451e-b7ae-3d8512366c59","Type":"ContainerStarted","Data":"c7256a75f7f86cccc9d8a466bc0ccfe17cae2269e9afbdb37ff7bdeed8f741fb"} Oct 29 23:02:42.318712 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:42.318638 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" podStartSLOduration=1.318617158 podStartE2EDuration="1.318617158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:42.281493435 +0000 UTC m=+1216.222041347" watchObservedRunningTime="2026-10-29 23:02:42.318617158 +0000 UTC m=+1216.259165069" Oct 29 23:02:42.710799 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:42.710766 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:43.711912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:43.711882 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 29 23:02:43.712351 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:43.711932 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" podUID="e778eb53-b230-451e-b7ae-3d8512366c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 29 23:02:44.710311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:44.710281 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 29 23:02:44.710496 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:44.710331 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" podUID="e778eb53-b230-451e-b7ae-3d8512366c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 29 23:02:44.911832 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:44.911796 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2"] Oct 29 23:02:44.940335 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:44.937465 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2" Oct 29 23:02:44.946115 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:44.946082 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2"] Oct 29 23:02:44.968329 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:44.968261 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c88955-dockercfg-hlnq4\"" Oct 29 23:02:44.968464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:44.968365 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c88955\"" Oct 29 23:02:45.009392 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:45.009351 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl"] Oct 29 23:02:45.013680 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:45.013661 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl" Oct 29 23:02:45.060718 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:45.051423 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15a298b4-75d8-4ca0-bad5-585c1e5c2058-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2\" (UID: \"15a298b4-75d8-4ca0-bad5-585c1e5c2058\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2" Oct 29 23:02:45.060718 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:45.051500 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6ztx\" (UniqueName: \"kubernetes.io/projected/15a298b4-75d8-4ca0-bad5-585c1e5c2058-kube-api-access-t6ztx\") pod \"payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2\" (UID: \"15a298b4-75d8-4ca0-bad5-585c1e5c2058\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2" Oct 29 23:02:45.086025 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:45.064088 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl"] Oct 29 23:02:45.287959 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:45.287524 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d561e85-cff7-461f-9504-7edcac13fb00-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl\" (UID: \"2d561e85-cff7-461f-9504-7edcac13fb00\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl" Oct 29 23:02:45.287959 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:45.287742 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15a298b4-75d8-4ca0-bad5-585c1e5c2058-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2\" (UID: \"15a298b4-75d8-4ca0-bad5-585c1e5c2058\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2" Oct 29 23:02:45.287959 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:45.287783 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sbv5\" (UniqueName: \"kubernetes.io/projected/2d561e85-cff7-461f-9504-7edcac13fb00-kube-api-access-9sbv5\") pod \"payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl\" (UID: \"2d561e85-cff7-461f-9504-7edcac13fb00\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl" Oct 29 23:02:45.287959 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:45.287843 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6ztx\" (UniqueName: \"kubernetes.io/projected/15a298b4-75d8-4ca0-bad5-585c1e5c2058-kube-api-access-t6ztx\") pod \"payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2\" (UID: \"15a298b4-75d8-4ca0-bad5-585c1e5c2058\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2" Oct 29 23:02:45.293662 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:45.293632 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15a298b4-75d8-4ca0-bad5-585c1e5c2058-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2\" (UID: \"15a298b4-75d8-4ca0-bad5-585c1e5c2058\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2" Oct 29 23:02:45.343344 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:45.343308 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6ztx\" (UniqueName: \"kubernetes.io/projected/15a298b4-75d8-4ca0-bad5-585c1e5c2058-kube-api-access-t6ztx\") pod \"payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2\" (UID: \"15a298b4-75d8-4ca0-bad5-585c1e5c2058\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2" Oct 29 23:02:45.392699 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:45.392662 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d561e85-cff7-461f-9504-7edcac13fb00-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl\" (UID: \"2d561e85-cff7-461f-9504-7edcac13fb00\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl" Oct 29 23:02:45.392917 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:45.392820 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sbv5\" (UniqueName: \"kubernetes.io/projected/2d561e85-cff7-461f-9504-7edcac13fb00-kube-api-access-9sbv5\") pod \"payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl\" (UID: \"2d561e85-cff7-461f-9504-7edcac13fb00\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl" Oct 29 23:02:45.393837 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:45.393808 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d561e85-cff7-461f-9504-7edcac13fb00-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl\" (UID: \"2d561e85-cff7-461f-9504-7edcac13fb00\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl" Oct 29 23:02:45.419326 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:45.419255 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sbv5\" (UniqueName: \"kubernetes.io/projected/2d561e85-cff7-461f-9504-7edcac13fb00-kube-api-access-9sbv5\") pod \"payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl\" (UID: \"2d561e85-cff7-461f-9504-7edcac13fb00\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl" Oct 29 23:02:45.561077 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:45.560991 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2" Oct 29 23:02:45.581420 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:45.581390 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl" Oct 29 23:02:45.710261 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:45.710231 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 29 23:02:45.710523 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:45.710470 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" podUID="e778eb53-b230-451e-b7ae-3d8512366c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 29 23:02:45.900982 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:45.900946 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2"] Oct 29 23:02:45.905335 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:02:45.905302 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a298b4_75d8_4ca0_bad5_585c1e5c2058.slice/crio-07c22400a3aa66562ce6872f56c1ed4a5beb455a260a982fb984381317980796 WatchSource:0}: Error finding container 07c22400a3aa66562ce6872f56c1ed4a5beb455a260a982fb984381317980796: Status 404 returned error can't find the container with id 07c22400a3aa66562ce6872f56c1ed4a5beb455a260a982fb984381317980796 Oct 29 23:02:45.952661 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:02:45.952632 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d561e85_cff7_461f_9504_7edcac13fb00.slice/crio-b5850f7ad40613255d782b29527da0f08a0b4c7208afa04471004c86944ba0f3 WatchSource:0}: Error finding container b5850f7ad40613255d782b29527da0f08a0b4c7208afa04471004c86944ba0f3: Status 404 returned error can't find the container with id b5850f7ad40613255d782b29527da0f08a0b4c7208afa04471004c86944ba0f3 Oct 29 23:02:45.953561 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:45.953482 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl"] Oct 29 23:02:46.286977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:46.286861 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl" event={"ID":"2d561e85-cff7-461f-9504-7edcac13fb00","Type":"ContainerStarted","Data":"50862a06dfc0784f5b14709f4747d1defb49149a164ecad5a75690db2a7dcba5"} Oct 29 23:02:46.287139 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:46.286986 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl" event={"ID":"2d561e85-cff7-461f-9504-7edcac13fb00","Type":"ContainerStarted","Data":"b5850f7ad40613255d782b29527da0f08a0b4c7208afa04471004c86944ba0f3"} Oct 29 23:02:46.287194 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:46.287136 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl" Oct 29 23:02:46.288976 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:46.288951 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2" event={"ID":"15a298b4-75d8-4ca0-bad5-585c1e5c2058","Type":"ContainerStarted","Data":"8e6c8b0185e5c77ac3d3af0a4f71b89bb9bb039dfbd8259da3022b4abd186735"} Oct 29 23:02:46.289126 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:46.288992 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2" event={"ID":"15a298b4-75d8-4ca0-bad5-585c1e5c2058","Type":"ContainerStarted","Data":"07c22400a3aa66562ce6872f56c1ed4a5beb455a260a982fb984381317980796"} Oct 29 23:02:46.289126 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:46.289052 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2" Oct 29 23:02:46.354453 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:46.354387 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl" podStartSLOduration=2.3543682329999998 podStartE2EDuration="2.354368233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:46.333683511 +0000 UTC m=+1220.274231422" watchObservedRunningTime="2026-10-29 23:02:46.354368233 +0000 UTC m=+1220.294916144" Oct 29 23:02:46.552591 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:46.552499 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-774bff-b8475597b-gkstk" podUID="789e0bd6-1808-413e-acc1-72537e22d107" containerName="payload-processing" containerID="cri-o://b6d1b65506a6fcc33dee7fbcac489d6dac2ab0a5ddfdf1659458995e5b72a438" gracePeriod=25 Oct 29 23:02:46.711248 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:46.711176 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 29 23:02:46.711248 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:46.711247 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" podUID="e778eb53-b230-451e-b7ae-3d8512366c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 29 23:02:46.880882 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:46.880857 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-774bff-b8475597b-gkstk" Oct 29 23:02:46.962409 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:46.962348 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2" podStartSLOduration=2.962329119 podStartE2EDuration="2.962329119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:46.391716744 +0000 UTC m=+1220.332264691" watchObservedRunningTime="2026-10-29 23:02:46.962329119 +0000 UTC m=+1220.902877030" Oct 29 23:02:47.044832 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:47.044797 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789e0bd6-1808-413e-acc1-72537e22d107-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789e0bd6-1808-413e-acc1-72537e22d107-plugins-config-volume\") pod \"789e0bd6-1808-413e-acc1-72537e22d107\" (UID: \"789e0bd6-1808-413e-acc1-72537e22d107\") " Oct 29 23:02:47.044987 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:47.044865 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789e0bd6-1808-413e-acc1-72537e22d107-kube-api-access-rv55z\" (UniqueName: \"kubernetes.io/projected/789e0bd6-1808-413e-acc1-72537e22d107-kube-api-access-rv55z\") pod \"789e0bd6-1808-413e-acc1-72537e22d107\" (UID: \"789e0bd6-1808-413e-acc1-72537e22d107\") " Oct 29 23:02:47.045189 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:47.045167 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789e0bd6-1808-413e-acc1-72537e22d107-plugins-config-volume" pod "789e0bd6-1808-413e-acc1-72537e22d107" (UID: "789e0bd6-1808-413e-acc1-72537e22d107"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:47.045432 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:47.045417 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789e0bd6-1808-413e-acc1-72537e22d107-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:47.047227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:47.047160 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789e0bd6-1808-413e-acc1-72537e22d107-kube-api-access-rv55z" pod "789e0bd6-1808-413e-acc1-72537e22d107" (UID: "789e0bd6-1808-413e-acc1-72537e22d107"). InnerVolumeSpecName "kube-api-access-rv55z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:47.147024 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:47.146966 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv55z\" (UniqueName: \"kubernetes.io/projected/789e0bd6-1808-413e-acc1-72537e22d107-kube-api-access-rv55z\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:02:47.299391 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:47.299351 4821 generic.go:356] "Generic (PLEG): container finished" podID="789e0bd6-1808-413e-acc1-72537e22d107" containerID="b6d1b65506a6fcc33dee7fbcac489d6dac2ab0a5ddfdf1659458995e5b72a438" exitCode=0 Oct 29 23:02:47.299569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:47.299410 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-774bff-b8475597b-gkstk" Oct 29 23:02:47.299569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:47.299427 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-774bff-b8475597b-gkstk" event={"ID":"789e0bd6-1808-413e-acc1-72537e22d107","Type":"ContainerDied","Data":"b6d1b65506a6fcc33dee7fbcac489d6dac2ab0a5ddfdf1659458995e5b72a438"} Oct 29 23:02:47.299569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:47.299478 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-774bff-b8475597b-gkstk" event={"ID":"789e0bd6-1808-413e-acc1-72537e22d107","Type":"ContainerDied","Data":"37730255a0f599bbc84302986cc4c79bd1143d927ddaf4cb57778577b5b6c88c"} Oct 29 23:02:47.299569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:47.299508 4821 scope.go:122] "RemoveContainer" containerID="b6d1b65506a6fcc33dee7fbcac489d6dac2ab0a5ddfdf1659458995e5b72a438" Oct 29 23:02:47.313271 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:47.313252 4821 scope.go:122] "RemoveContainer" containerID="b6d1b65506a6fcc33dee7fbcac489d6dac2ab0a5ddfdf1659458995e5b72a438" Oct 29 23:02:47.313626 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:02:47.313607 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6d1b65506a6fcc33dee7fbcac489d6dac2ab0a5ddfdf1659458995e5b72a438\": container with ID starting with b6d1b65506a6fcc33dee7fbcac489d6dac2ab0a5ddfdf1659458995e5b72a438 not found: ID does not exist" containerID="b6d1b65506a6fcc33dee7fbcac489d6dac2ab0a5ddfdf1659458995e5b72a438" Oct 29 23:02:47.313724 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:47.313634 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6d1b65506a6fcc33dee7fbcac489d6dac2ab0a5ddfdf1659458995e5b72a438"} err="failed to get container status \"b6d1b65506a6fcc33dee7fbcac489d6dac2ab0a5ddfdf1659458995e5b72a438\": rpc error: code = NotFound desc = could not find container \"b6d1b65506a6fcc33dee7fbcac489d6dac2ab0a5ddfdf1659458995e5b72a438\": container with ID starting with b6d1b65506a6fcc33dee7fbcac489d6dac2ab0a5ddfdf1659458995e5b72a438 not found: ID does not exist" Oct 29 23:02:47.361119 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:47.361085 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-774bff-b8475597b-gkstk"] Oct 29 23:02:47.366925 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:47.366899 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-774bff-b8475597b-gkstk"] Oct 29 23:02:47.710880 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:47.710844 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 29 23:02:47.711070 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:47.710919 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" podUID="e778eb53-b230-451e-b7ae-3d8512366c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 29 23:02:48.182382 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:48.182339 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="789e0bd6-1808-413e-acc1-72537e22d107" path="/var/lib/kubelet/pods/789e0bd6-1808-413e-acc1-72537e22d107/volumes" Oct 29 23:02:48.714618 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:48.714589 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:48.715124 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:48.715103 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:48.715917 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:48.715894 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:02:50.202433 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:50.202404 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g" Oct 29 23:02:50.202857 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:50.202697 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn" Oct 29 23:02:52.230888 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:52.230859 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz" Oct 29 23:02:52.232889 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:52.232869 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp" Oct 29 23:02:53.888497 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:02:53.888463 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:02:57.302840 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:57.302804 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl" Oct 29 23:02:57.303303 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:02:57.302953 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2" Oct 29 23:03:11.432368 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.432331 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2"] Oct 29 23:03:11.433586 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.433554 4821 cpu_manager.go:414] "containerMap: removing container" podUID="789e0bd6-1808-413e-acc1-72537e22d107" containerName="payload-processing" Oct 29 23:03:11.433586 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.433578 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789e0bd6-1808-413e-acc1-72537e22d107" containerName="payload-processing" Oct 29 23:03:11.433778 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.433690 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="789e0bd6-1808-413e-acc1-72537e22d107" containerName="payload-processing" Oct 29 23:03:11.438767 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.438702 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.473562 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.473527 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.473749 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.473686 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.473819 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.473792 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g4k4\" (UniqueName: \"kubernetes.io/projected/dcc509cf-1d6b-4453-b46b-d756cd4ec293-kube-api-access-4g4k4\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.473934 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.473861 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.473998 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.473974 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.474055 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.474017 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.474178 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.474161 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.474307 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.474291 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.474378 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.474342 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.491276 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.489453 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2"] Oct 29 23:03:11.539599 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.539568 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552"] Oct 29 23:03:11.544949 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.544915 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.565020 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.563580 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552"] Oct 29 23:03:11.584244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.582674 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.584244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.582771 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g4k4\" (UniqueName: \"kubernetes.io/projected/dcc509cf-1d6b-4453-b46b-d756cd4ec293-kube-api-access-4g4k4\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.584244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.582841 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.584244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.582918 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.584244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.582946 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.584244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.583082 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.584244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.583145 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.584244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.583216 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.584244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.583283 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.584244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.583815 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.584244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.584069 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.584244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.584090 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.585037 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.584312 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.585466 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.585412 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.606382 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.597985 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.606382 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.598555 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.610681 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.610175 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.619596 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.619512 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g4k4\" (UniqueName: \"kubernetes.io/projected/dcc509cf-1d6b-4453-b46b-d756cd4ec293-kube-api-access-4g4k4\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.684359 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.684276 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.684569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.684448 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.684569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.684513 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39efdaf8-97f8-47fc-964c-4bc0daab505f-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.684687 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.684571 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkgsj\" (UniqueName: \"kubernetes.io/projected/39efdaf8-97f8-47fc-964c-4bc0daab505f-kube-api-access-bkgsj\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.684747 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.684689 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.684747 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.684721 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.684848 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.684746 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.684848 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.684809 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.684848 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.684835 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.757396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.757364 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:11.786770 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.786743 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.786904 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.786873 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.786976 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.786921 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39efdaf8-97f8-47fc-964c-4bc0daab505f-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.786976 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.786939 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkgsj\" (UniqueName: \"kubernetes.io/projected/39efdaf8-97f8-47fc-964c-4bc0daab505f-kube-api-access-bkgsj\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.787092 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.787004 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.787092 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.787028 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.787092 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.787051 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.787092 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.787086 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.787365 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.787124 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.787365 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.787301 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-workload-certs\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.787608 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.787573 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-credential-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.787787 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.787659 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39efdaf8-97f8-47fc-964c-4bc0daab505f-istiod-ca-cert\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.787926 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.787885 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-data\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.788152 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.788132 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-workload-socket\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.789749 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.789722 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-envoy\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.790048 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.790030 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-podinfo\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.814529 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.813737 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-token\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.814529 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.813847 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkgsj\" (UniqueName: \"kubernetes.io/projected/39efdaf8-97f8-47fc-964c-4bc0daab505f-kube-api-access-bkgsj\") pod \"e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:11.886914 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:11.886880 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:12.096265 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:03:12.096229 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc509cf_1d6b_4453_b46b_d756cd4ec293.slice/crio-28155a18a4507ef7074323931ea8e1b740dcd3554536809c87f6ead7ec6889ff WatchSource:0}: Error finding container 28155a18a4507ef7074323931ea8e1b740dcd3554536809c87f6ead7ec6889ff: Status 404 returned error can't find the container with id 28155a18a4507ef7074323931ea8e1b740dcd3554536809c87f6ead7ec6889ff Oct 29 23:03:12.096372 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:12.096325 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2"] Oct 29 23:03:12.098899 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:12.098869 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:03:12.098988 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:12.098930 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:03:12.098988 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:12.098960 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:03:12.211324 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:12.211298 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552"] Oct 29 23:03:12.212907 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:03:12.212876 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39efdaf8_97f8_47fc_964c_4bc0daab505f.slice/crio-af6f522d6cee952edab2b1a636f049859af2041f8c21ebc1956bfc2bbde9ad04 WatchSource:0}: Error finding container af6f522d6cee952edab2b1a636f049859af2041f8c21ebc1956bfc2bbde9ad04: Status 404 returned error can't find the container with id af6f522d6cee952edab2b1a636f049859af2041f8c21ebc1956bfc2bbde9ad04 Oct 29 23:03:12.215919 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:12.215881 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:03:12.216039 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:12.215952 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:03:12.216039 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:12.215999 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:03:12.574348 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:12.574241 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" event={"ID":"39efdaf8-97f8-47fc-964c-4bc0daab505f","Type":"ContainerStarted","Data":"0fe29c8f2b6d2d9654843ba173b17de1a3b6508e9a4dd2242b6b420d8e026557"} Oct 29 23:03:12.574348 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:12.574316 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" event={"ID":"39efdaf8-97f8-47fc-964c-4bc0daab505f","Type":"ContainerStarted","Data":"af6f522d6cee952edab2b1a636f049859af2041f8c21ebc1956bfc2bbde9ad04"} Oct 29 23:03:12.576371 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:12.576341 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" event={"ID":"dcc509cf-1d6b-4453-b46b-d756cd4ec293","Type":"ContainerStarted","Data":"6082b264f6820cb7d0dcf63f2b1d01654d05fafd24bbb5e11ddadf89ca45f503"} Oct 29 23:03:12.576509 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:12.576388 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" event={"ID":"dcc509cf-1d6b-4453-b46b-d756cd4ec293","Type":"ContainerStarted","Data":"28155a18a4507ef7074323931ea8e1b740dcd3554536809c87f6ead7ec6889ff"} Oct 29 23:03:12.695877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:12.695822 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" podStartSLOduration=1.695803012 podStartE2EDuration="1.695803012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:12.647160588 +0000 UTC m=+1246.587708501" watchObservedRunningTime="2026-10-29 23:03:12.695803012 +0000 UTC m=+1246.636350922" Oct 29 23:03:12.696638 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:12.696577 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" podStartSLOduration=1.696564574 podStartE2EDuration="1.696564574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:12.695050906 +0000 UTC m=+1246.635598815" watchObservedRunningTime="2026-10-29 23:03:12.696564574 +0000 UTC m=+1246.637112488" Oct 29 23:03:12.762222 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:12.761311 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:12.887764 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:12.887732 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:13.764168 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:13.764055 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 29 23:03:13.764168 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:13.764121 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" podUID="dcc509cf-1d6b-4453-b46b-d756cd4ec293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 29 23:03:13.889289 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:13.889249 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 29 23:03:13.889480 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:13.889303 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" podUID="39efdaf8-97f8-47fc-964c-4bc0daab505f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 29 23:03:14.599739 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:14.599709 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz"] Oct 29 23:03:14.604012 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:14.603996 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz" Oct 29 23:03:14.662226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:14.655648 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:03:14.662226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:14.655749 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:03:14.678764 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:14.674960 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz"] Oct 29 23:03:14.803242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:14.787479 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6895c1ac-8ff5-4caf-b82a-f729ed0c4cff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz\" (UID: \"6895c1ac-8ff5-4caf-b82a-f729ed0c4cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz" Oct 29 23:03:14.803242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:14.787612 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6895c1ac-8ff5-4caf-b82a-f729ed0c4cff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz\" (UID: \"6895c1ac-8ff5-4caf-b82a-f729ed0c4cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz" Oct 29 23:03:14.803242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:14.787688 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqx2t\" (UniqueName: \"kubernetes.io/projected/6895c1ac-8ff5-4caf-b82a-f729ed0c4cff-kube-api-access-rqx2t\") pod \"maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz\" (UID: \"6895c1ac-8ff5-4caf-b82a-f729ed0c4cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz" Oct 29 23:03:14.803242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:14.802832 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 29 23:03:14.803242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:14.802889 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" podUID="dcc509cf-1d6b-4453-b46b-d756cd4ec293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 29 23:03:14.890239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:14.890172 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6895c1ac-8ff5-4caf-b82a-f729ed0c4cff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz\" (UID: \"6895c1ac-8ff5-4caf-b82a-f729ed0c4cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz" Oct 29 23:03:14.890448 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:14.890303 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqx2t\" (UniqueName: \"kubernetes.io/projected/6895c1ac-8ff5-4caf-b82a-f729ed0c4cff-kube-api-access-rqx2t\") pod \"maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz\" (UID: \"6895c1ac-8ff5-4caf-b82a-f729ed0c4cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz" Oct 29 23:03:14.890862 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:14.890837 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6895c1ac-8ff5-4caf-b82a-f729ed0c4cff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz\" (UID: \"6895c1ac-8ff5-4caf-b82a-f729ed0c4cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz" Oct 29 23:03:14.891981 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:14.891931 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6895c1ac-8ff5-4caf-b82a-f729ed0c4cff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz\" (UID: \"6895c1ac-8ff5-4caf-b82a-f729ed0c4cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz" Oct 29 23:03:14.892874 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:14.892844 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 29 23:03:14.892986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:14.892899 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" podUID="39efdaf8-97f8-47fc-964c-4bc0daab505f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 29 23:03:14.900154 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:14.899967 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6895c1ac-8ff5-4caf-b82a-f729ed0c4cff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz\" (UID: \"6895c1ac-8ff5-4caf-b82a-f729ed0c4cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz" Oct 29 23:03:14.981675 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:14.981638 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqx2t\" (UniqueName: \"kubernetes.io/projected/6895c1ac-8ff5-4caf-b82a-f729ed0c4cff-kube-api-access-rqx2t\") pod \"maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz\" (UID: \"6895c1ac-8ff5-4caf-b82a-f729ed0c4cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz" Oct 29 23:03:15.247749 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:15.247655 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz" Oct 29 23:03:15.674874 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:15.674841 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz"] Oct 29 23:03:15.762812 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:15.762769 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 29 23:03:15.763173 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:15.763136 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" podUID="dcc509cf-1d6b-4453-b46b-d756cd4ec293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 29 23:03:15.888467 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:15.888426 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 29 23:03:15.888958 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:15.888482 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" podUID="39efdaf8-97f8-47fc-964c-4bc0daab505f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 29 23:03:16.610733 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:16.610669 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz" event={"ID":"6895c1ac-8ff5-4caf-b82a-f729ed0c4cff","Type":"ContainerStarted","Data":"2593eb33b190e716633ab15496c8f44429c57f5517f453a816423e20831d7abd"} Oct 29 23:03:16.610733 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:16.610736 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz" event={"ID":"6895c1ac-8ff5-4caf-b82a-f729ed0c4cff","Type":"ContainerStarted","Data":"4d4841a5ed13754e7f71c71565e2ad5d50ada2c05c770b5796721e31d003e714"} Oct 29 23:03:16.764225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:16.760592 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 29 23:03:16.764225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:16.760655 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" podUID="dcc509cf-1d6b-4453-b46b-d756cd4ec293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 29 23:03:16.909501 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:16.909404 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 29 23:03:16.909501 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:16.909476 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" podUID="39efdaf8-97f8-47fc-964c-4bc0daab505f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 29 23:03:17.666225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:17.665938 4821 generic.go:356] "Generic (PLEG): container finished" podID="6895c1ac-8ff5-4caf-b82a-f729ed0c4cff" containerID="2593eb33b190e716633ab15496c8f44429c57f5517f453a816423e20831d7abd" exitCode=0 Oct 29 23:03:17.666225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:17.665994 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz" event={"ID":"6895c1ac-8ff5-4caf-b82a-f729ed0c4cff","Type":"ContainerDied","Data":"2593eb33b190e716633ab15496c8f44429c57f5517f453a816423e20831d7abd"} Oct 29 23:03:17.759625 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:17.759588 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 29 23:03:17.759940 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:17.759907 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" podUID="dcc509cf-1d6b-4453-b46b-d756cd4ec293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 29 23:03:17.887759 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:17.887723 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 29 23:03:17.887937 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:17.887774 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" podUID="39efdaf8-97f8-47fc-964c-4bc0daab505f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 29 23:03:18.764185 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:18.764151 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:18.765038 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:18.764890 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:18.765946 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:18.765922 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:03:18.837638 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:18.837611 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz" Oct 29 23:03:18.894007 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:18.893980 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:18.895381 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:18.895357 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:18.896284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:18.896259 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:03:18.896691 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:18.896668 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6895c1ac-8ff5-4caf-b82a-f729ed0c4cff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6895c1ac-8ff5-4caf-b82a-f729ed0c4cff-maas-api-xks-ca\") pod \"6895c1ac-8ff5-4caf-b82a-f729ed0c4cff\" (UID: \"6895c1ac-8ff5-4caf-b82a-f729ed0c4cff\") " Oct 29 23:03:18.896780 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:18.896734 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6895c1ac-8ff5-4caf-b82a-f729ed0c4cff-kube-api-access-rqx2t\" (UniqueName: \"kubernetes.io/projected/6895c1ac-8ff5-4caf-b82a-f729ed0c4cff-kube-api-access-rqx2t\") pod \"6895c1ac-8ff5-4caf-b82a-f729ed0c4cff\" (UID: \"6895c1ac-8ff5-4caf-b82a-f729ed0c4cff\") " Oct 29 23:03:18.896914 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:18.896901 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6895c1ac-8ff5-4caf-b82a-f729ed0c4cff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6895c1ac-8ff5-4caf-b82a-f729ed0c4cff-maas-api-openshift-ca\") pod \"6895c1ac-8ff5-4caf-b82a-f729ed0c4cff\" (UID: \"6895c1ac-8ff5-4caf-b82a-f729ed0c4cff\") " Oct 29 23:03:18.898219 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:18.898180 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6895c1ac-8ff5-4caf-b82a-f729ed0c4cff-maas-api-openshift-ca" pod "6895c1ac-8ff5-4caf-b82a-f729ed0c4cff" (UID: "6895c1ac-8ff5-4caf-b82a-f729ed0c4cff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:18.900133 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:18.900108 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6895c1ac-8ff5-4caf-b82a-f729ed0c4cff-kube-api-access-rqx2t" pod "6895c1ac-8ff5-4caf-b82a-f729ed0c4cff" (UID: "6895c1ac-8ff5-4caf-b82a-f729ed0c4cff"). InnerVolumeSpecName "kube-api-access-rqx2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:18.900447 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:18.900421 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6895c1ac-8ff5-4caf-b82a-f729ed0c4cff-maas-api-xks-ca" pod "6895c1ac-8ff5-4caf-b82a-f729ed0c4cff" (UID: "6895c1ac-8ff5-4caf-b82a-f729ed0c4cff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:19.000146 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:19.000065 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6895c1ac-8ff5-4caf-b82a-f729ed0c4cff-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:03:19.000640 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:19.000618 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6895c1ac-8ff5-4caf-b82a-f729ed0c4cff-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:03:19.000876 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:19.000859 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqx2t\" (UniqueName: \"kubernetes.io/projected/6895c1ac-8ff5-4caf-b82a-f729ed0c4cff-kube-api-access-rqx2t\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:03:19.684342 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:19.684268 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz" Oct 29 23:03:19.684502 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:19.684263 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz" event={"ID":"6895c1ac-8ff5-4caf-b82a-f729ed0c4cff","Type":"ContainerDied","Data":"4d4841a5ed13754e7f71c71565e2ad5d50ada2c05c770b5796721e31d003e714"} Oct 29 23:03:19.684502 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:19.684449 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d4841a5ed13754e7f71c71565e2ad5d50ada2c05c770b5796721e31d003e714" Oct 29 23:03:23.887633 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:03:23.887592 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:03:24.183574 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.183479 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4"] Oct 29 23:03:24.184167 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.184145 4821 cpu_manager.go:414] "containerMap: removing container" podUID="6895c1ac-8ff5-4caf-b82a-f729ed0c4cff" containerName="revoke-keys" Oct 29 23:03:24.184167 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.184162 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6895c1ac-8ff5-4caf-b82a-f729ed0c4cff" containerName="revoke-keys" Oct 29 23:03:24.184385 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.184374 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="6895c1ac-8ff5-4caf-b82a-f729ed0c4cff" containerName="revoke-keys" Oct 29 23:03:24.193073 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.193045 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.216329 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.216298 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4"] Oct 29 23:03:24.223878 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.221150 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-14debc-openshift-default-dockercfg-bm4w8\"" Oct 29 23:03:24.320495 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.320453 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-workload-socket\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.338610 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.338518 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-envoy\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.339216 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.339171 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-token\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.339476 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.339458 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-data\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.355543 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.355495 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istiod-ca-cert\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.355930 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.355873 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-workload-certs\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.356236 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.356179 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4dbb\" (UniqueName: \"kubernetes.io/projected/d654f3f3-8c6b-46e4-b528-0459173f9d2a-kube-api-access-l4dbb\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.356454 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.356439 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-podinfo\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.356655 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.356641 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-credential-socket\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.482348 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.482243 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-workload-socket\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.483163 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.483139 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-envoy\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.483800 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.483777 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-token\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.484377 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.484354 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-data\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.484556 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.484540 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istiod-ca-cert\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.484697 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.484682 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-workload-certs\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.484900 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.484884 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4dbb\" (UniqueName: \"kubernetes.io/projected/d654f3f3-8c6b-46e4-b528-0459173f9d2a-kube-api-access-l4dbb\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.485037 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.485022 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-podinfo\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.485174 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.485159 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-credential-socket\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.485715 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.485692 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-credential-socket\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.485846 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.482992 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-workload-socket\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.486220 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.486179 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-data\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.487220 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.487143 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istiod-ca-cert\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.487615 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.487593 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-workload-certs\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.521226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.520012 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-token\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.521226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.520617 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4dbb\" (UniqueName: \"kubernetes.io/projected/d654f3f3-8c6b-46e4-b528-0459173f9d2a-kube-api-access-l4dbb\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.521226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.520903 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-podinfo\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.538818 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.535478 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-envoy\") pod \"e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:24.830633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:24.830546 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:25.200378 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:25.200345 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4"] Oct 29 23:03:25.201659 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:03:25.201632 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd654f3f3_8c6b_46e4_b528_0459173f9d2a.slice/crio-16b6af43a8fdfb3599076a1c741a980160f6f3fd34f9fea263810ccfb5931416 WatchSource:0}: Error finding container 16b6af43a8fdfb3599076a1c741a980160f6f3fd34f9fea263810ccfb5931416: Status 404 returned error can't find the container with id 16b6af43a8fdfb3599076a1c741a980160f6f3fd34f9fea263810ccfb5931416 Oct 29 23:03:25.205105 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:25.205062 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:03:25.205192 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:25.205164 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:03:25.205280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:25.205234 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:03:25.741014 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:25.740967 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" event={"ID":"d654f3f3-8c6b-46e4-b528-0459173f9d2a","Type":"ContainerStarted","Data":"77c8d1d1c29d955bf8d7e8e1b2df5a3fad1b59108a158739d1ccadd269372b55"} Oct 29 23:03:25.741248 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:25.741040 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" event={"ID":"d654f3f3-8c6b-46e4-b528-0459173f9d2a","Type":"ContainerStarted","Data":"16b6af43a8fdfb3599076a1c741a980160f6f3fd34f9fea263810ccfb5931416"} Oct 29 23:03:25.832225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:25.830744 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:26.667832 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:26.667796 4821 scope.go:122] "RemoveContainer" containerID="2c62bcd60a13bf986c6f8325e18a619a2cad7337921c410b4c4d888ad82cd8a3" Oct 29 23:03:26.837421 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:26.837389 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:26.921055 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:26.920958 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" podStartSLOduration=2.920945126 podStartE2EDuration="2.920945126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:25.826565366 +0000 UTC m=+1259.767113276" watchObservedRunningTime="2026-10-29 23:03:26.920945126 +0000 UTC m=+1260.861493103" Oct 29 23:03:27.787223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:27.783694 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:27.791238 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:27.790450 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:03:29.686335 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:29.686285 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:29.895226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:29.894975 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 23:03:29.959842 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:29.959740 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:29.989225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:29.987286 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:29.989225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:29.987691 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 23:03:30.621238 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:30.621089 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:30.829577 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:30.829513 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27"] Oct 29 23:03:30.897336 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:30.896758 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:30.957227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:30.848638 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 23:03:30.985227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:30.960701 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 29 23:03:31.060227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.057766 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 29 23:03:31.125229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.074364 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 29 23:03:31.125229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.075050 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 23:03:31.207456 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.180228 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 23:03:31.207456 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.201370 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27"] Oct 29 23:03:31.207456 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.201533 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 23:03:31.207456 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.201595 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 23:03:31.207456 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.201648 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 23:03:31.207456 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.192859 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27" Oct 29 23:03:31.214011 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.211850 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 23:03:31.282227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.267037 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-14debc-dockercfg-mzlq4\"" Oct 29 23:03:31.282227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.267435 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 23:03:31.336284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.310325 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h4hj\" (UniqueName: \"kubernetes.io/projected/0ae12788-66ec-4e5f-9248-00df150bba93-kube-api-access-4h4hj\") pod \"payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27\" (UID: \"0ae12788-66ec-4e5f-9248-00df150bba93\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27" Oct 29 23:03:31.338126 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.310938 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-14debc\"" Oct 29 23:03:31.338426 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.338394 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae12788-66ec-4e5f-9248-00df150bba93-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27\" (UID: \"0ae12788-66ec-4e5f-9248-00df150bba93\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27" Oct 29 23:03:31.406335 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.395779 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:31.410352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.410322 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 23:03:31.488674 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.488582 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae12788-66ec-4e5f-9248-00df150bba93-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27\" (UID: \"0ae12788-66ec-4e5f-9248-00df150bba93\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27" Oct 29 23:03:31.489225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.489180 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h4hj\" (UniqueName: \"kubernetes.io/projected/0ae12788-66ec-4e5f-9248-00df150bba93-kube-api-access-4h4hj\") pod \"payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27\" (UID: \"0ae12788-66ec-4e5f-9248-00df150bba93\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27" Oct 29 23:03:31.505781 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.495729 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae12788-66ec-4e5f-9248-00df150bba93-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27\" (UID: \"0ae12788-66ec-4e5f-9248-00df150bba93\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27" Oct 29 23:03:31.506263 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.506232 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:31.550226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.549834 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 23:03:31.560053 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.560015 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h4hj\" (UniqueName: \"kubernetes.io/projected/0ae12788-66ec-4e5f-9248-00df150bba93-kube-api-access-4h4hj\") pod \"payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27\" (UID: \"0ae12788-66ec-4e5f-9248-00df150bba93\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27" Oct 29 23:03:31.710228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.706619 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27" Oct 29 23:03:31.969229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.966549 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt"] Oct 29 23:03:31.969229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:31.966911 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt" podUID="07038c7b-89e8-4830-949b-a0755eec2c3a" containerName="maas-api" containerID="cri-o://b593a5ae390b1af7a95d036f9bea3b62e84b4bf61ed11502be2dd946947f0af7" gracePeriod=30 Oct 29 23:03:32.708289 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:32.707313 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:33.282227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:33.281949 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:33.323322 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:33.318316 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-774bff-r77mm"] Oct 29 23:03:33.392232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:33.391312 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:33.392773 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:33.392695 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:33.409441 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:33.399673 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:33.411601 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:33.410632 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:33.462441 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:33.462403 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:33.495560 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:33.494924 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-774bff-r77mm"] Oct 29 23:03:33.609396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:33.609356 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:33.684481 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:33.680337 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:33.694800 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:33.694752 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27"] Oct 29 23:03:33.696072 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:33.696033 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 29 23:03:33.710871 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:33.710829 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 23:03:33.812747 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:33.812710 4821 generic.go:356] "Generic (PLEG): container finished" podID="07038c7b-89e8-4830-949b-a0755eec2c3a" containerID="b593a5ae390b1af7a95d036f9bea3b62e84b4bf61ed11502be2dd946947f0af7" exitCode=0 Oct 29 23:03:33.813176 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:33.812889 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt" event={"ID":"07038c7b-89e8-4830-949b-a0755eec2c3a","Type":"ContainerDied","Data":"b593a5ae390b1af7a95d036f9bea3b62e84b4bf61ed11502be2dd946947f0af7"} Oct 29 23:03:33.853894 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:33.853799 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded" Oct 29 23:03:33.885506 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:33.865978 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt" Oct 29 23:03:34.005388 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:34.005350 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07038c7b-89e8-4830-949b-a0755eec2c3a-kube-api-access-7z7rq\" (UniqueName: \"kubernetes.io/projected/07038c7b-89e8-4830-949b-a0755eec2c3a-kube-api-access-7z7rq\") pod \"07038c7b-89e8-4830-949b-a0755eec2c3a\" (UID: \"07038c7b-89e8-4830-949b-a0755eec2c3a\") " Oct 29 23:03:34.005558 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:34.005408 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07038c7b-89e8-4830-949b-a0755eec2c3a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07038c7b-89e8-4830-949b-a0755eec2c3a-maas-api-tls\") pod \"07038c7b-89e8-4830-949b-a0755eec2c3a\" (UID: \"07038c7b-89e8-4830-949b-a0755eec2c3a\") " Oct 29 23:03:34.005558 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:34.005507 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07038c7b-89e8-4830-949b-a0755eec2c3a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07038c7b-89e8-4830-949b-a0755eec2c3a-openshift-service-ca\") pod \"07038c7b-89e8-4830-949b-a0755eec2c3a\" (UID: \"07038c7b-89e8-4830-949b-a0755eec2c3a\") " Oct 29 23:03:34.005787 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:34.005744 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07038c7b-89e8-4830-949b-a0755eec2c3a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/07038c7b-89e8-4830-949b-a0755eec2c3a-metrics-certs\") pod \"07038c7b-89e8-4830-949b-a0755eec2c3a\" (UID: \"07038c7b-89e8-4830-949b-a0755eec2c3a\") " Oct 29 23:03:34.009972 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:34.009942 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07038c7b-89e8-4830-949b-a0755eec2c3a-openshift-service-ca" pod "07038c7b-89e8-4830-949b-a0755eec2c3a" (UID: "07038c7b-89e8-4830-949b-a0755eec2c3a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:34.017304 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:34.014320 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07038c7b-89e8-4830-949b-a0755eec2c3a-metrics-certs" pod "07038c7b-89e8-4830-949b-a0755eec2c3a" (UID: "07038c7b-89e8-4830-949b-a0755eec2c3a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:34.097301 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:34.095465 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07038c7b-89e8-4830-949b-a0755eec2c3a-maas-api-tls" pod "07038c7b-89e8-4830-949b-a0755eec2c3a" (UID: "07038c7b-89e8-4830-949b-a0755eec2c3a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:34.097301 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:34.096365 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07038c7b-89e8-4830-949b-a0755eec2c3a-kube-api-access-7z7rq" pod "07038c7b-89e8-4830-949b-a0755eec2c3a" (UID: "07038c7b-89e8-4830-949b-a0755eec2c3a"). InnerVolumeSpecName "kube-api-access-7z7rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:34.108719 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:34.108692 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07038c7b-89e8-4830-949b-a0755eec2c3a-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:03:34.108903 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:34.108890 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z7rq\" (UniqueName: \"kubernetes.io/projected/07038c7b-89e8-4830-949b-a0755eec2c3a-kube-api-access-7z7rq\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:03:34.108985 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:34.108975 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07038c7b-89e8-4830-949b-a0755eec2c3a-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:03:34.109077 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:34.109066 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07038c7b-89e8-4830-949b-a0755eec2c3a-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:03:34.300742 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:34.300703 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecaa3b06-01ff-4bf1-b709-47d221d32de0" path="/var/lib/kubelet/pods/ecaa3b06-01ff-4bf1-b709-47d221d32de0/volumes" Oct 29 23:03:34.918168 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:34.918141 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt" Oct 29 23:03:34.935169 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:34.918492 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt" event={"ID":"07038c7b-89e8-4830-949b-a0755eec2c3a","Type":"ContainerDied","Data":"15d5e62e04eb03c5d2c4f4336d0aed316c41c866f7aaf73faa1d46a44b0ae613"} Oct 29 23:03:34.935400 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:34.935215 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27" event={"ID":"0ae12788-66ec-4e5f-9248-00df150bba93","Type":"ContainerStarted","Data":"4db4e9d4c95111ef3c390ae80c3373d03529f8319ded017eb00779aeae21508a"} Oct 29 23:03:34.935400 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:34.935261 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27" event={"ID":"0ae12788-66ec-4e5f-9248-00df150bba93","Type":"ContainerStarted","Data":"b56c0ffafe7d0b1af99cc916775ea4bfa8ba05ba1ab5de8252b1684023838ade"} Oct 29 23:03:34.935400 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:34.935289 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27" Oct 29 23:03:34.935400 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:34.935325 4821 scope.go:122] "RemoveContainer" containerID="b593a5ae390b1af7a95d036f9bea3b62e84b4bf61ed11502be2dd946947f0af7" Oct 29 23:03:35.000264 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:34.998434 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27" podStartSLOduration=4.998414476 podStartE2EDuration="4.998414476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:34.981717461 +0000 UTC m=+1268.922265372" watchObservedRunningTime="2026-10-29 23:03:34.998414476 +0000 UTC m=+1268.938962388" Oct 29 23:03:35.028484 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:35.028446 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt"] Oct 29 23:03:35.039789 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:35.039753 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-774bff-f779d4d66-2tmwt"] Oct 29 23:03:36.167490 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:36.167158 4821 patch_prober.go:29] interesting pod/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:03:36.167490 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:36.167287 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" podUID="9e08c58b-702a-437f-bb87-1a4bc4727f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:36.182009 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:36.181974 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07038c7b-89e8-4830-949b-a0755eec2c3a" path="/var/lib/kubelet/pods/07038c7b-89e8-4830-949b-a0755eec2c3a/volumes" Oct 29 23:03:37.188395 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:37.188349 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:03:37.188870 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:37.188422 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" podUID="1e845eec-edd8-4ce2-ad4c-fbfc15fead43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:40.325384 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:40.325349 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj"] Oct 29 23:03:40.325945 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:40.325650 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" podUID="59c2911d-7186-44a7-9b7b-e5c55f1c185a" containerName="istio-proxy" containerID="cri-o://e06b4d41461dc76cc60571ca0c826e5e8b45a1e93126572c4085192a39ef9f5c" gracePeriod=30 Oct 29 23:03:42.065230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.064179 4821 generic.go:356] "Generic (PLEG): container finished" podID="9e08c58b-702a-437f-bb87-1a4bc4727f78" containerID="159bb96543d6f0d3c192fa136d44fbc179e9692be2614277723c82383d25b5fa" exitCode=0 Oct 29 23:03:42.065230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.064358 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" event={"ID":"9e08c58b-702a-437f-bb87-1a4bc4727f78","Type":"ContainerDied","Data":"159bb96543d6f0d3c192fa136d44fbc179e9692be2614277723c82383d25b5fa"} Oct 29 23:03:42.065230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.064890 4821 scope.go:122] "RemoveContainer" containerID="159bb96543d6f0d3c192fa136d44fbc179e9692be2614277723c82383d25b5fa" Oct 29 23:03:42.068777 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.068288 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:03:42.068777 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.068386 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:03:42.068777 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.068463 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:03:42.517869 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.517783 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd"] Oct 29 23:03:42.518519 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.518491 4821 cpu_manager.go:414] "containerMap: removing container" podUID="07038c7b-89e8-4830-949b-a0755eec2c3a" containerName="maas-api" Oct 29 23:03:42.518519 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.518512 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07038c7b-89e8-4830-949b-a0755eec2c3a" containerName="maas-api" Oct 29 23:03:42.518698 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.518645 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="07038c7b-89e8-4830-949b-a0755eec2c3a" containerName="maas-api" Oct 29 23:03:42.529119 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.529090 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.545853 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.545795 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd"] Oct 29 23:03:42.572231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.572058 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-1b7950-openshift-default-dockercfg-pqvgz\"" Oct 29 23:03:42.642139 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.642089 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-workload-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.642696 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.642656 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-credential-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.642876 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.642859 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-workload-certs\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.643081 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.643065 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-podinfo\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.643271 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.643256 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istiod-ca-cert\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.643522 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.643506 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-data\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.643749 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.643732 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-envoy\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.643937 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.643922 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb5cg\" (UniqueName: \"kubernetes.io/projected/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-kube-api-access-zb5cg\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.644102 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.644087 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-token\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.749352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.749313 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-workload-certs\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.760570 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.760509 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-workload-certs\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.774415 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.760865 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-podinfo\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.774415 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.774394 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istiod-ca-cert\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.774659 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.774535 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-data\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.774659 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.774630 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-envoy\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.774783 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.774694 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb5cg\" (UniqueName: \"kubernetes.io/projected/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-kube-api-access-zb5cg\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.774783 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.774743 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-token\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.774900 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.774843 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-workload-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.775123 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.775101 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-credential-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.775643 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.775616 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-credential-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.793227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.781043 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-data\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.817572 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.783220 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-workload-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.817572 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.785959 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-envoy\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.817572 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.786969 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-podinfo\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.840726 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.840632 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istiod-ca-cert\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.851909 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.851847 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb5cg\" (UniqueName: \"kubernetes.io/projected/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-kube-api-access-zb5cg\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.855392 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.855364 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-token\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:42.861169 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:42.860983 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:43.153833 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:43.151367 4821 generic.go:356] "Generic (PLEG): container finished" podID="1e845eec-edd8-4ce2-ad4c-fbfc15fead43" containerID="0a7dc024e82fada74a07789f502359da954a9573b26b37e5715885b2ee9f6fd4" exitCode=0 Oct 29 23:03:43.153833 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:43.151597 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" event={"ID":"1e845eec-edd8-4ce2-ad4c-fbfc15fead43","Type":"ContainerDied","Data":"0a7dc024e82fada74a07789f502359da954a9573b26b37e5715885b2ee9f6fd4"} Oct 29 23:03:43.153833 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:43.152183 4821 scope.go:122] "RemoveContainer" containerID="0a7dc024e82fada74a07789f502359da954a9573b26b37e5715885b2ee9f6fd4" Oct 29 23:03:43.154824 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:43.154525 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:03:43.154824 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:43.154629 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:03:43.154824 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:43.154695 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:03:43.161837 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:43.156642 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" event={"ID":"9e08c58b-702a-437f-bb87-1a4bc4727f78","Type":"ContainerStarted","Data":"f005b9644e366d8b8a2b67408b681da59e52d607ee2e3396ea815013bca3a01b"} Oct 29 23:03:43.367909 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:43.367811 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd"] Oct 29 23:03:43.450138 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:43.449319 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:03:43.450138 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:43.449421 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:03:43.450138 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:43.449463 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:03:44.010147 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:44.010054 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:03:44.011414 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:44.011360 4821 patch_prober.go:29] interesting pod/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 29 23:03:44.011566 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:44.011446 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" podUID="9e08c58b-702a-437f-bb87-1a4bc4727f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 29 23:03:44.254310 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:44.253471 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" event={"ID":"1e845eec-edd8-4ce2-ad4c-fbfc15fead43","Type":"ContainerStarted","Data":"b4f66bf9c50adf7317b266ccd161958452cd8fada3efab7ef9697b413d10270c"} Oct 29 23:03:44.258953 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:44.257911 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" event={"ID":"f1840a05-a8fe-40e2-b7f9-39a91f6f355f","Type":"ContainerStarted","Data":"dc0623074d766ff1f978853f514e08c7f9386d8f86c3c52ce1f735aba831b7be"} Oct 29 23:03:44.258953 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:44.257969 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" event={"ID":"f1840a05-a8fe-40e2-b7f9-39a91f6f355f","Type":"ContainerStarted","Data":"4a422e96f5832d6e516a5611088b7bb6880495807d5fb408e166bc23488f6b94"} Oct 29 23:03:44.278214 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:44.261698 4821 generic.go:356] "Generic (PLEG): container finished" podID="83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" containerID="fc92a35acaaa8e1ede0400b1dbc674112d76faec8ed5c23a2b0eb72e0bba1893" exitCode=0 Oct 29 23:03:44.278214 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:44.262766 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" event={"ID":"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10","Type":"ContainerDied","Data":"fc92a35acaaa8e1ede0400b1dbc674112d76faec8ed5c23a2b0eb72e0bba1893"} Oct 29 23:03:44.278214 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:44.263242 4821 scope.go:122] "RemoveContainer" containerID="fc92a35acaaa8e1ede0400b1dbc674112d76faec8ed5c23a2b0eb72e0bba1893" Oct 29 23:03:44.283543 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:44.283507 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:03:44.283684 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:44.283599 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:03:44.283684 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:44.283647 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:03:44.506423 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:44.506079 4821 patch_prober.go:29] interesting pod/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:03:44.506423 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:44.506151 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" podUID="59c2911d-7186-44a7-9b7b-e5c55f1c185a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:03:44.863223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:44.862555 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:44.890227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:44.889706 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:44.951230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:44.946474 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" podStartSLOduration=2.946450598 podStartE2EDuration="2.946450598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:44.503586873 +0000 UTC m=+1278.444134783" watchObservedRunningTime="2026-10-29 23:03:44.946450598 +0000 UTC m=+1278.886998510" Oct 29 23:03:44.998407 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:44.998374 4821 patch_prober.go:29] interesting pod/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 29 23:03:44.998657 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:44.998636 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" podUID="9e08c58b-702a-437f-bb87-1a4bc4727f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 29 23:03:45.044311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:45.044266 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4"] Oct 29 23:03:45.124677 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:45.124586 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4" Oct 29 23:03:45.144234 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:45.142759 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:03:45.144234 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:45.142903 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 29 23:03:45.144234 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:45.142962 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" podUID="1e845eec-edd8-4ce2-ad4c-fbfc15fead43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 29 23:03:45.153487 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:45.153445 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:03:45.156119 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:45.154256 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:03:45.418288 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:45.356879 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1d1479f-84c6-46d1-b27c-65fa46cb9fcc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4\" (UID: \"b1d1479f-84c6-46d1-b27c-65fa46cb9fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4" Oct 29 23:03:45.419466 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:45.419430 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1d1479f-84c6-46d1-b27c-65fa46cb9fcc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4\" (UID: \"b1d1479f-84c6-46d1-b27c-65fa46cb9fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4" Oct 29 23:03:45.419652 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:45.419629 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh989\" (UniqueName: \"kubernetes.io/projected/b1d1479f-84c6-46d1-b27c-65fa46cb9fcc-kube-api-access-fh989\") pod \"maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4\" (UID: \"b1d1479f-84c6-46d1-b27c-65fa46cb9fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4" Oct 29 23:03:45.421713 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:45.421691 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4"] Oct 29 23:03:45.512087 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:45.512045 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" event={"ID":"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10","Type":"ContainerStarted","Data":"979a10477c756ce5f6c1a1c6fbd1bc2b7616a3b5c12789e0f11ad78f78c36f3e"} Oct 29 23:03:45.514160 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:45.513077 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:45.541228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:45.522694 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1d1479f-84c6-46d1-b27c-65fa46cb9fcc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4\" (UID: \"b1d1479f-84c6-46d1-b27c-65fa46cb9fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4" Oct 29 23:03:45.541666 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:45.541627 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1d1479f-84c6-46d1-b27c-65fa46cb9fcc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4\" (UID: \"b1d1479f-84c6-46d1-b27c-65fa46cb9fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4" Oct 29 23:03:45.541851 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:45.541831 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh989\" (UniqueName: \"kubernetes.io/projected/b1d1479f-84c6-46d1-b27c-65fa46cb9fcc-kube-api-access-fh989\") pod \"maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4\" (UID: \"b1d1479f-84c6-46d1-b27c-65fa46cb9fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4" Oct 29 23:03:45.544132 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:45.544103 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1d1479f-84c6-46d1-b27c-65fa46cb9fcc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4\" (UID: \"b1d1479f-84c6-46d1-b27c-65fa46cb9fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4" Oct 29 23:03:45.548288 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:45.544322 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:03:45.583531 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:45.583499 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh989\" (UniqueName: \"kubernetes.io/projected/b1d1479f-84c6-46d1-b27c-65fa46cb9fcc-kube-api-access-fh989\") pod \"maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4\" (UID: \"b1d1479f-84c6-46d1-b27c-65fa46cb9fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4" Oct 29 23:03:45.585618 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:45.585592 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1d1479f-84c6-46d1-b27c-65fa46cb9fcc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4\" (UID: \"b1d1479f-84c6-46d1-b27c-65fa46cb9fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4" Oct 29 23:03:45.788335 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:45.787987 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4" Oct 29 23:03:45.997654 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:45.997618 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27" Oct 29 23:03:46.000419 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:45.999988 4821 patch_prober.go:29] interesting pod/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 29 23:03:46.000419 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.000068 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" podUID="9e08c58b-702a-437f-bb87-1a4bc4727f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 29 23:03:46.078149 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.077256 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:03:46.109975 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.109349 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 29 23:03:46.109975 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.109409 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" podUID="83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 29 23:03:46.187272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.186953 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 29 23:03:46.187272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.187021 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" podUID="1e845eec-edd8-4ce2-ad4c-fbfc15fead43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 29 23:03:46.572229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.562416 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:03:46.574386 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.574350 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-credential-socket\") pod \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " Oct 29 23:03:46.574658 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.574636 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-workload-certs\") pod \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " Oct 29 23:03:46.583367 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.583333 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-workload-certs" pod "59c2911d-7186-44a7-9b7b-e5c55f1c185a" (UID: "59c2911d-7186-44a7-9b7b-e5c55f1c185a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:46.602398 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.602353 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-credential-socket" pod "59c2911d-7186-44a7-9b7b-e5c55f1c185a" (UID: "59c2911d-7186-44a7-9b7b-e5c55f1c185a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:46.692439 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.680932 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-envoy\") pod \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " Oct 29 23:03:46.692439 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.681002 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-podinfo\") pod \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " Oct 29 23:03:46.692439 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.681121 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-workload-socket\") pod \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " Oct 29 23:03:46.692439 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.681163 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istiod-ca-cert\") pod \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " Oct 29 23:03:46.692439 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.681448 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c2911d-7186-44a7-9b7b-e5c55f1c185a-kube-api-access-pb48p\" (UniqueName: \"kubernetes.io/projected/59c2911d-7186-44a7-9b7b-e5c55f1c185a-kube-api-access-pb48p\") pod \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " Oct 29 23:03:46.692439 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.681523 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-data\") pod \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " Oct 29 23:03:46.692439 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.681603 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-token\" (UniqueName: \"kubernetes.io/projected/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-token\") pod \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " Oct 29 23:03:46.696246 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.692981 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:03:46.696246 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.693012 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:03:46.711841 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.711783 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-workload-socket" pod "59c2911d-7186-44a7-9b7b-e5c55f1c185a" (UID: "59c2911d-7186-44a7-9b7b-e5c55f1c185a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:46.809471 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.809436 4821 generic.go:356] "Generic (PLEG): container finished" podID="59c2911d-7186-44a7-9b7b-e5c55f1c185a" containerID="e06b4d41461dc76cc60571ca0c826e5e8b45a1e93126572c4085192a39ef9f5c" exitCode=0 Oct 29 23:03:46.810853 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.810834 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" Oct 29 23:03:46.811432 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.811410 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" event={"ID":"59c2911d-7186-44a7-9b7b-e5c55f1c185a","Type":"ContainerDied","Data":"e06b4d41461dc76cc60571ca0c826e5e8b45a1e93126572c4085192a39ef9f5c"} Oct 29 23:03:46.811610 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.811593 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj" event={"ID":"59c2911d-7186-44a7-9b7b-e5c55f1c185a","Type":"ContainerDied","Data":"24a62afd074ba78643ea9bc1e73218c12d17b4321d22dcfa3e1b49871a0adafb"} Oct 29 23:03:46.811736 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.811723 4821 scope.go:122] "RemoveContainer" containerID="e06b4d41461dc76cc60571ca0c826e5e8b45a1e93126572c4085192a39ef9f5c" Oct 29 23:03:46.813444 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.812721 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istiod-ca-cert" pod "59c2911d-7186-44a7-9b7b-e5c55f1c185a" (UID: "59c2911d-7186-44a7-9b7b-e5c55f1c185a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:46.813444 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.812972 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istiod-ca-cert\") pod \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\" (UID: \"59c2911d-7186-44a7-9b7b-e5c55f1c185a\") " Oct 29 23:03:46.823255 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:03:46.813727 4821 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/59c2911d-7186-44a7-9b7b-e5c55f1c185a/volumes/kubernetes.io~configmap/istiod-ca-cert Oct 29 23:03:46.823255 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.813740 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:03:46.823255 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.813745 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istiod-ca-cert" pod "59c2911d-7186-44a7-9b7b-e5c55f1c185a" (UID: "59c2911d-7186-44a7-9b7b-e5c55f1c185a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:46.852347 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.852306 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-data" pod "59c2911d-7186-44a7-9b7b-e5c55f1c185a" (UID: "59c2911d-7186-44a7-9b7b-e5c55f1c185a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:46.967327 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.916888 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:03:46.967327 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.916915 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:03:46.985908 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.985849 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-token" pod "59c2911d-7186-44a7-9b7b-e5c55f1c185a" (UID: "59c2911d-7186-44a7-9b7b-e5c55f1c185a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:46.992784 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.992731 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-envoy" pod "59c2911d-7186-44a7-9b7b-e5c55f1c185a" (UID: "59c2911d-7186-44a7-9b7b-e5c55f1c185a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:47.001093 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.999295 4821 patch_prober.go:29] interesting pod/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 29 23:03:47.001093 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:46.999352 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" podUID="9e08c58b-702a-437f-bb87-1a4bc4727f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 29 23:03:47.008724 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:47.007462 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-podinfo" pod "59c2911d-7186-44a7-9b7b-e5c55f1c185a" (UID: "59c2911d-7186-44a7-9b7b-e5c55f1c185a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:03:47.012086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:47.011463 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c2911d-7186-44a7-9b7b-e5c55f1c185a-kube-api-access-pb48p" pod "59c2911d-7186-44a7-9b7b-e5c55f1c185a" (UID: "59c2911d-7186-44a7-9b7b-e5c55f1c185a"). InnerVolumeSpecName "kube-api-access-pb48p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:47.079578 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:47.037419 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:03:47.079578 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:47.037453 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:03:47.079578 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:47.037467 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c2911d-7186-44a7-9b7b-e5c55f1c185a-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:03:47.079578 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:47.037480 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb48p\" (UniqueName: \"kubernetes.io/projected/59c2911d-7186-44a7-9b7b-e5c55f1c185a-kube-api-access-pb48p\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:03:47.079578 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:47.042130 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 29 23:03:47.079578 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:47.042224 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" podUID="83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 29 23:03:47.096467 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:47.084358 4821 scope.go:122] "RemoveContainer" containerID="9cd89555453f6592bcd7e640bbbe6c16d378d123539af7cda8219930f3338018" Oct 29 23:03:47.199085 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:47.146924 4821 scope.go:122] "RemoveContainer" containerID="e06b4d41461dc76cc60571ca0c826e5e8b45a1e93126572c4085192a39ef9f5c" Oct 29 23:03:47.199085 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:03:47.196443 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e06b4d41461dc76cc60571ca0c826e5e8b45a1e93126572c4085192a39ef9f5c\": container with ID starting with e06b4d41461dc76cc60571ca0c826e5e8b45a1e93126572c4085192a39ef9f5c not found: ID does not exist" containerID="e06b4d41461dc76cc60571ca0c826e5e8b45a1e93126572c4085192a39ef9f5c" Oct 29 23:03:47.199085 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:47.196490 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e06b4d41461dc76cc60571ca0c826e5e8b45a1e93126572c4085192a39ef9f5c"} err="failed to get container status \"e06b4d41461dc76cc60571ca0c826e5e8b45a1e93126572c4085192a39ef9f5c\": rpc error: code = NotFound desc = could not find container \"e06b4d41461dc76cc60571ca0c826e5e8b45a1e93126572c4085192a39ef9f5c\": container with ID starting with e06b4d41461dc76cc60571ca0c826e5e8b45a1e93126572c4085192a39ef9f5c not found: ID does not exist" Oct 29 23:03:47.199085 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:47.196517 4821 scope.go:122] "RemoveContainer" containerID="9cd89555453f6592bcd7e640bbbe6c16d378d123539af7cda8219930f3338018" Oct 29 23:03:47.199085 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:03:47.197331 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cd89555453f6592bcd7e640bbbe6c16d378d123539af7cda8219930f3338018\": container with ID starting with 9cd89555453f6592bcd7e640bbbe6c16d378d123539af7cda8219930f3338018 not found: ID does not exist" containerID="9cd89555453f6592bcd7e640bbbe6c16d378d123539af7cda8219930f3338018" Oct 29 23:03:47.199085 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:47.197366 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cd89555453f6592bcd7e640bbbe6c16d378d123539af7cda8219930f3338018"} err="failed to get container status \"9cd89555453f6592bcd7e640bbbe6c16d378d123539af7cda8219930f3338018\": rpc error: code = NotFound desc = could not find container \"9cd89555453f6592bcd7e640bbbe6c16d378d123539af7cda8219930f3338018\": container with ID starting with 9cd89555453f6592bcd7e640bbbe6c16d378d123539af7cda8219930f3338018 not found: ID does not exist" Oct 29 23:03:47.199760 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:47.199739 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 29 23:03:47.199977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:47.199961 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" podUID="1e845eec-edd8-4ce2-ad4c-fbfc15fead43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 29 23:03:47.313382 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:47.313351 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj"] Oct 29 23:03:47.517258 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:47.515680 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4"] Oct 29 23:03:48.036501 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:48.036467 4821 patch_prober.go:29] interesting pod/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 29 23:03:48.050127 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:48.050075 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" podUID="9e08c58b-702a-437f-bb87-1a4bc4727f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 29 23:03:48.097345 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:48.097306 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-774bff-openshift-default-68cd6447f-7h5qj"] Oct 29 23:03:48.111499 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:48.111393 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 29 23:03:48.200109 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:48.174284 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4" event={"ID":"b1d1479f-84c6-46d1-b27c-65fa46cb9fcc","Type":"ContainerStarted","Data":"9205ef3545f78db920fbff7e610ff651d30f94218b3d6f302760a55f79222bf4"} Oct 29 23:03:48.200693 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:48.200115 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" podUID="83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 29 23:03:48.200888 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:48.199405 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 29 23:03:48.201009 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:48.200989 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" podUID="1e845eec-edd8-4ce2-ad4c-fbfc15fead43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 29 23:03:48.211847 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:48.211821 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59c2911d-7186-44a7-9b7b-e5c55f1c185a" path="/var/lib/kubelet/pods/59c2911d-7186-44a7-9b7b-e5c55f1c185a/volumes" Oct 29 23:03:48.998874 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:48.998837 4821 patch_prober.go:29] interesting pod/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 29 23:03:48.999057 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:48.998899 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" podUID="9e08c58b-702a-437f-bb87-1a4bc4727f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 29 23:03:49.048666 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:49.048628 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 29 23:03:49.049307 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:49.049274 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" podUID="83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 29 23:03:49.103431 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:49.103391 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:03:49.133228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:49.127768 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 29 23:03:49.133228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:49.127829 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" podUID="1e845eec-edd8-4ce2-ad4c-fbfc15fead43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 29 23:03:49.310870 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:49.310335 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4" event={"ID":"b1d1479f-84c6-46d1-b27c-65fa46cb9fcc","Type":"ContainerStarted","Data":"6e43f16535746b5e36916ee3cc736adc6917561e04fd094fb24df0228cfb6394"} Oct 29 23:03:49.376261 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:49.374059 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4" podStartSLOduration=5.374037723 podStartE2EDuration="5.374037723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:49.367708619 +0000 UTC m=+1283.308256530" watchObservedRunningTime="2026-10-29 23:03:49.374037723 +0000 UTC m=+1283.314585651" Oct 29 23:03:50.047228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:50.043550 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 29 23:03:50.047228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:50.043612 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" podUID="83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 29 23:03:50.117227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:50.116350 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:03:50.132905 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:50.132871 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:03:50.212943 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:50.212530 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 29 23:03:50.212943 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:50.212590 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" podUID="1e845eec-edd8-4ce2-ad4c-fbfc15fead43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 29 23:03:50.370056 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:50.370019 4821 generic.go:356] "Generic (PLEG): container finished" podID="b1d1479f-84c6-46d1-b27c-65fa46cb9fcc" containerID="6e43f16535746b5e36916ee3cc736adc6917561e04fd094fb24df0228cfb6394" exitCode=0 Oct 29 23:03:50.370269 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:50.370123 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4" event={"ID":"b1d1479f-84c6-46d1-b27c-65fa46cb9fcc","Type":"ContainerDied","Data":"6e43f16535746b5e36916ee3cc736adc6917561e04fd094fb24df0228cfb6394"} Oct 29 23:03:50.371561 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:50.371539 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:03:51.069225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:51.062236 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 29 23:03:51.069225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:51.062302 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" podUID="83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 29 23:03:51.147222 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:51.142541 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 29 23:03:51.147222 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:51.142599 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" podUID="1e845eec-edd8-4ce2-ad4c-fbfc15fead43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 29 23:03:51.147851 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:51.147639 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:03:51.776252 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:51.776141 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl"] Oct 29 23:03:51.784531 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:51.783260 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2"] Oct 29 23:03:51.876300 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:51.873324 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4" Oct 29 23:03:51.966188 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:51.966151 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1d1479f-84c6-46d1-b27c-65fa46cb9fcc-kube-api-access-fh989\" (UniqueName: \"kubernetes.io/projected/b1d1479f-84c6-46d1-b27c-65fa46cb9fcc-kube-api-access-fh989\") pod \"b1d1479f-84c6-46d1-b27c-65fa46cb9fcc\" (UID: \"b1d1479f-84c6-46d1-b27c-65fa46cb9fcc\") " Oct 29 23:03:51.966376 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:51.966362 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1d1479f-84c6-46d1-b27c-65fa46cb9fcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1d1479f-84c6-46d1-b27c-65fa46cb9fcc-maas-api-xks-ca\") pod \"b1d1479f-84c6-46d1-b27c-65fa46cb9fcc\" (UID: \"b1d1479f-84c6-46d1-b27c-65fa46cb9fcc\") " Oct 29 23:03:51.966596 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:51.966572 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1d1479f-84c6-46d1-b27c-65fa46cb9fcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1d1479f-84c6-46d1-b27c-65fa46cb9fcc-maas-api-openshift-ca\") pod \"b1d1479f-84c6-46d1-b27c-65fa46cb9fcc\" (UID: \"b1d1479f-84c6-46d1-b27c-65fa46cb9fcc\") " Oct 29 23:03:51.967091 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:51.967042 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1d1479f-84c6-46d1-b27c-65fa46cb9fcc-maas-api-openshift-ca" pod "b1d1479f-84c6-46d1-b27c-65fa46cb9fcc" (UID: "b1d1479f-84c6-46d1-b27c-65fa46cb9fcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:51.967314 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:51.967295 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1d1479f-84c6-46d1-b27c-65fa46cb9fcc-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:03:51.972527 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:51.972499 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1d1479f-84c6-46d1-b27c-65fa46cb9fcc-maas-api-xks-ca" pod "b1d1479f-84c6-46d1-b27c-65fa46cb9fcc" (UID: "b1d1479f-84c6-46d1-b27c-65fa46cb9fcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:51.972725 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:51.972696 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1d1479f-84c6-46d1-b27c-65fa46cb9fcc-kube-api-access-fh989" pod "b1d1479f-84c6-46d1-b27c-65fa46cb9fcc" (UID: "b1d1479f-84c6-46d1-b27c-65fa46cb9fcc"). InnerVolumeSpecName "kube-api-access-fh989". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:52.076314 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:52.075429 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 29 23:03:52.076314 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:52.075503 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" podUID="83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 29 23:03:52.106158 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:52.085765 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh989\" (UniqueName: \"kubernetes.io/projected/b1d1479f-84c6-46d1-b27c-65fa46cb9fcc-kube-api-access-fh989\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:03:52.106158 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:52.085802 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1d1479f-84c6-46d1-b27c-65fa46cb9fcc-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:03:52.329517 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:52.329369 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:03:52.337072 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:52.337040 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:03:52.417650 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:52.417618 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4" Oct 29 23:03:52.417858 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:52.417648 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4" event={"ID":"b1d1479f-84c6-46d1-b27c-65fa46cb9fcc","Type":"ContainerDied","Data":"9205ef3545f78db920fbff7e610ff651d30f94218b3d6f302760a55f79222bf4"} Oct 29 23:03:52.417858 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:52.417698 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9205ef3545f78db920fbff7e610ff651d30f94218b3d6f302760a55f79222bf4" Oct 29 23:03:53.045431 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:53.045402 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:03:53.046487 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:53.046462 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:03:53.886563 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:03:53.886522 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:03:55.636775 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:55.636744 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs"] Oct 29 23:03:55.828192 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:55.828156 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz"] Oct 29 23:03:55.832808 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:55.832779 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f173b1-9lzcz"] Oct 29 23:03:56.175908 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:56.175876 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6895c1ac-8ff5-4caf-b82a-f729ed0c4cff" path="/var/lib/kubelet/pods/6895c1ac-8ff5-4caf-b82a-f729ed0c4cff/volumes" Oct 29 23:03:56.776795 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:56.776743 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl" podUID="2d561e85-cff7-461f-9504-7edcac13fb00" containerName="payload-processing" containerID="cri-o://50862a06dfc0784f5b14709f4747d1defb49149a164ecad5a75690db2a7dcba5" gracePeriod=25 Oct 29 23:03:56.785491 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:56.785448 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2" podUID="15a298b4-75d8-4ca0-bad5-585c1e5c2058" containerName="payload-pre-processing" containerID="cri-o://8e6c8b0185e5c77ac3d3af0a4f71b89bb9bb039dfbd8259da3022b4abd186735" gracePeriod=25 Oct 29 23:03:57.120280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.119901 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl" Oct 29 23:03:57.158696 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.158664 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2" Oct 29 23:03:57.161686 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.161657 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d561e85-cff7-461f-9504-7edcac13fb00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d561e85-cff7-461f-9504-7edcac13fb00-plugins-config-volume\") pod \"2d561e85-cff7-461f-9504-7edcac13fb00\" (UID: \"2d561e85-cff7-461f-9504-7edcac13fb00\") " Oct 29 23:03:57.162023 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.162000 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d561e85-cff7-461f-9504-7edcac13fb00-plugins-config-volume" pod "2d561e85-cff7-461f-9504-7edcac13fb00" (UID: "2d561e85-cff7-461f-9504-7edcac13fb00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:57.162116 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.162083 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d561e85-cff7-461f-9504-7edcac13fb00-kube-api-access-9sbv5\" (UniqueName: \"kubernetes.io/projected/2d561e85-cff7-461f-9504-7edcac13fb00-kube-api-access-9sbv5\") pod \"2d561e85-cff7-461f-9504-7edcac13fb00\" (UID: \"2d561e85-cff7-461f-9504-7edcac13fb00\") " Oct 29 23:03:57.162808 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.162783 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d561e85-cff7-461f-9504-7edcac13fb00-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:03:57.167633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.165603 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d561e85-cff7-461f-9504-7edcac13fb00-kube-api-access-9sbv5" pod "2d561e85-cff7-461f-9504-7edcac13fb00" (UID: "2d561e85-cff7-461f-9504-7edcac13fb00"). InnerVolumeSpecName "kube-api-access-9sbv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:57.264045 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.264007 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15a298b4-75d8-4ca0-bad5-585c1e5c2058-kube-api-access-t6ztx\" (UniqueName: \"kubernetes.io/projected/15a298b4-75d8-4ca0-bad5-585c1e5c2058-kube-api-access-t6ztx\") pod \"15a298b4-75d8-4ca0-bad5-585c1e5c2058\" (UID: \"15a298b4-75d8-4ca0-bad5-585c1e5c2058\") " Oct 29 23:03:57.264270 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.264108 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15a298b4-75d8-4ca0-bad5-585c1e5c2058-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15a298b4-75d8-4ca0-bad5-585c1e5c2058-plugins-config-volume\") pod \"15a298b4-75d8-4ca0-bad5-585c1e5c2058\" (UID: \"15a298b4-75d8-4ca0-bad5-585c1e5c2058\") " Oct 29 23:03:57.264485 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.264463 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a298b4-75d8-4ca0-bad5-585c1e5c2058-plugins-config-volume" pod "15a298b4-75d8-4ca0-bad5-585c1e5c2058" (UID: "15a298b4-75d8-4ca0-bad5-585c1e5c2058"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:57.264823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.264801 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15a298b4-75d8-4ca0-bad5-585c1e5c2058-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:03:57.264886 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.264829 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sbv5\" (UniqueName: \"kubernetes.io/projected/2d561e85-cff7-461f-9504-7edcac13fb00-kube-api-access-9sbv5\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:03:57.266272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.266249 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a298b4-75d8-4ca0-bad5-585c1e5c2058-kube-api-access-t6ztx" pod "15a298b4-75d8-4ca0-bad5-585c1e5c2058" (UID: "15a298b4-75d8-4ca0-bad5-585c1e5c2058"). InnerVolumeSpecName "kube-api-access-t6ztx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:57.366637 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.366606 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6ztx\" (UniqueName: \"kubernetes.io/projected/15a298b4-75d8-4ca0-bad5-585c1e5c2058-kube-api-access-t6ztx\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:03:57.610514 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.610477 4821 generic.go:356] "Generic (PLEG): container finished" podID="2d561e85-cff7-461f-9504-7edcac13fb00" containerID="50862a06dfc0784f5b14709f4747d1defb49149a164ecad5a75690db2a7dcba5" exitCode=0 Oct 29 23:03:57.610723 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.610581 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl" event={"ID":"2d561e85-cff7-461f-9504-7edcac13fb00","Type":"ContainerDied","Data":"50862a06dfc0784f5b14709f4747d1defb49149a164ecad5a75690db2a7dcba5"} Oct 29 23:03:57.610723 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.610626 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl" event={"ID":"2d561e85-cff7-461f-9504-7edcac13fb00","Type":"ContainerDied","Data":"b5850f7ad40613255d782b29527da0f08a0b4c7208afa04471004c86944ba0f3"} Oct 29 23:03:57.610723 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.610647 4821 scope.go:122] "RemoveContainer" containerID="50862a06dfc0784f5b14709f4747d1defb49149a164ecad5a75690db2a7dcba5" Oct 29 23:03:57.610849 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.610771 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl" Oct 29 23:03:57.616308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.616274 4821 generic.go:356] "Generic (PLEG): container finished" podID="15a298b4-75d8-4ca0-bad5-585c1e5c2058" containerID="8e6c8b0185e5c77ac3d3af0a4f71b89bb9bb039dfbd8259da3022b4abd186735" exitCode=0 Oct 29 23:03:57.616707 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.616628 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2" event={"ID":"15a298b4-75d8-4ca0-bad5-585c1e5c2058","Type":"ContainerDied","Data":"8e6c8b0185e5c77ac3d3af0a4f71b89bb9bb039dfbd8259da3022b4abd186735"} Oct 29 23:03:57.616813 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.616737 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2" event={"ID":"15a298b4-75d8-4ca0-bad5-585c1e5c2058","Type":"ContainerDied","Data":"07c22400a3aa66562ce6872f56c1ed4a5beb455a260a982fb984381317980796"} Oct 29 23:03:57.616873 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.616850 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2" Oct 29 23:03:57.644359 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.644267 4821 scope.go:122] "RemoveContainer" containerID="50862a06dfc0784f5b14709f4747d1defb49149a164ecad5a75690db2a7dcba5" Oct 29 23:03:57.646161 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:03:57.644810 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50862a06dfc0784f5b14709f4747d1defb49149a164ecad5a75690db2a7dcba5\": container with ID starting with 50862a06dfc0784f5b14709f4747d1defb49149a164ecad5a75690db2a7dcba5 not found: ID does not exist" containerID="50862a06dfc0784f5b14709f4747d1defb49149a164ecad5a75690db2a7dcba5" Oct 29 23:03:57.646161 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.644849 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50862a06dfc0784f5b14709f4747d1defb49149a164ecad5a75690db2a7dcba5"} err="failed to get container status \"50862a06dfc0784f5b14709f4747d1defb49149a164ecad5a75690db2a7dcba5\": rpc error: code = NotFound desc = could not find container \"50862a06dfc0784f5b14709f4747d1defb49149a164ecad5a75690db2a7dcba5\": container with ID starting with 50862a06dfc0784f5b14709f4747d1defb49149a164ecad5a75690db2a7dcba5 not found: ID does not exist" Oct 29 23:03:57.646161 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.644873 4821 scope.go:122] "RemoveContainer" containerID="8e6c8b0185e5c77ac3d3af0a4f71b89bb9bb039dfbd8259da3022b4abd186735" Oct 29 23:03:57.660033 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.659999 4821 scope.go:122] "RemoveContainer" containerID="8e6c8b0185e5c77ac3d3af0a4f71b89bb9bb039dfbd8259da3022b4abd186735" Oct 29 23:03:57.660430 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:03:57.660402 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e6c8b0185e5c77ac3d3af0a4f71b89bb9bb039dfbd8259da3022b4abd186735\": container with ID starting with 8e6c8b0185e5c77ac3d3af0a4f71b89bb9bb039dfbd8259da3022b4abd186735 not found: ID does not exist" containerID="8e6c8b0185e5c77ac3d3af0a4f71b89bb9bb039dfbd8259da3022b4abd186735" Oct 29 23:03:57.660526 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.660440 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e6c8b0185e5c77ac3d3af0a4f71b89bb9bb039dfbd8259da3022b4abd186735"} err="failed to get container status \"8e6c8b0185e5c77ac3d3af0a4f71b89bb9bb039dfbd8259da3022b4abd186735\": rpc error: code = NotFound desc = could not find container \"8e6c8b0185e5c77ac3d3af0a4f71b89bb9bb039dfbd8259da3022b4abd186735\": container with ID starting with 8e6c8b0185e5c77ac3d3af0a4f71b89bb9bb039dfbd8259da3022b4abd186735 not found: ID does not exist" Oct 29 23:03:57.690621 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.690568 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl"] Oct 29 23:03:57.702938 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.702906 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c88955-8b77b7b58-fnspl"] Oct 29 23:03:57.734588 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.734552 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2"] Oct 29 23:03:57.738981 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:57.738954 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c88955-6fcc4666cf-f7th2"] Oct 29 23:03:58.172437 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:58.172403 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15a298b4-75d8-4ca0-bad5-585c1e5c2058" path="/var/lib/kubelet/pods/15a298b4-75d8-4ca0-bad5-585c1e5c2058/volumes" Oct 29 23:03:58.172896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:03:58.172880 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d561e85-cff7-461f-9504-7edcac13fb00" path="/var/lib/kubelet/pods/2d561e85-cff7-461f-9504-7edcac13fb00/volumes" Oct 29 23:04:00.637877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:00.637837 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs" podUID="3c7c4b21-7397-4db3-b170-aaded0958eaa" containerName="payload-processing" containerID="cri-o://701a35a9df18d5b8eaebc2c0e79a51e7c183b8707fbaf3a99d1dbda478a7cc57" gracePeriod=25 Oct 29 23:04:00.914597 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:00.914563 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs" Oct 29 23:04:01.040256 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:01.040193 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c7c4b21-7397-4db3-b170-aaded0958eaa-kube-api-access-bhwcn\" (UniqueName: \"kubernetes.io/projected/3c7c4b21-7397-4db3-b170-aaded0958eaa-kube-api-access-bhwcn\") pod \"3c7c4b21-7397-4db3-b170-aaded0958eaa\" (UID: \"3c7c4b21-7397-4db3-b170-aaded0958eaa\") " Oct 29 23:04:01.040567 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:01.040552 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c7c4b21-7397-4db3-b170-aaded0958eaa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c7c4b21-7397-4db3-b170-aaded0958eaa-plugins-config-volume\") pod \"3c7c4b21-7397-4db3-b170-aaded0958eaa\" (UID: \"3c7c4b21-7397-4db3-b170-aaded0958eaa\") " Oct 29 23:04:01.040979 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:01.040959 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c7c4b21-7397-4db3-b170-aaded0958eaa-plugins-config-volume" pod "3c7c4b21-7397-4db3-b170-aaded0958eaa" (UID: "3c7c4b21-7397-4db3-b170-aaded0958eaa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:01.041303 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:01.041288 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c7c4b21-7397-4db3-b170-aaded0958eaa-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:01.042507 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:01.042483 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c7c4b21-7397-4db3-b170-aaded0958eaa-kube-api-access-bhwcn" pod "3c7c4b21-7397-4db3-b170-aaded0958eaa" (UID: "3c7c4b21-7397-4db3-b170-aaded0958eaa"). InnerVolumeSpecName "kube-api-access-bhwcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:01.142849 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:01.142775 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhwcn\" (UniqueName: \"kubernetes.io/projected/3c7c4b21-7397-4db3-b170-aaded0958eaa-kube-api-access-bhwcn\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:01.655300 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:01.655268 4821 generic.go:356] "Generic (PLEG): container finished" podID="3c7c4b21-7397-4db3-b170-aaded0958eaa" containerID="701a35a9df18d5b8eaebc2c0e79a51e7c183b8707fbaf3a99d1dbda478a7cc57" exitCode=0 Oct 29 23:04:01.655755 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:01.655339 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs" Oct 29 23:04:01.655755 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:01.655354 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs" event={"ID":"3c7c4b21-7397-4db3-b170-aaded0958eaa","Type":"ContainerDied","Data":"701a35a9df18d5b8eaebc2c0e79a51e7c183b8707fbaf3a99d1dbda478a7cc57"} Oct 29 23:04:01.655755 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:01.655436 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs" event={"ID":"3c7c4b21-7397-4db3-b170-aaded0958eaa","Type":"ContainerDied","Data":"5a19fa9c41e7c5af70c4b2bd2065133215d220263d800195d2c1903d8dddb009"} Oct 29 23:04:01.655755 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:01.655464 4821 scope.go:122] "RemoveContainer" containerID="701a35a9df18d5b8eaebc2c0e79a51e7c183b8707fbaf3a99d1dbda478a7cc57" Oct 29 23:04:01.668077 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:01.668057 4821 scope.go:122] "RemoveContainer" containerID="701a35a9df18d5b8eaebc2c0e79a51e7c183b8707fbaf3a99d1dbda478a7cc57" Oct 29 23:04:01.668402 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:04:01.668376 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"701a35a9df18d5b8eaebc2c0e79a51e7c183b8707fbaf3a99d1dbda478a7cc57\": container with ID starting with 701a35a9df18d5b8eaebc2c0e79a51e7c183b8707fbaf3a99d1dbda478a7cc57 not found: ID does not exist" containerID="701a35a9df18d5b8eaebc2c0e79a51e7c183b8707fbaf3a99d1dbda478a7cc57" Oct 29 23:04:01.668508 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:01.668409 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"701a35a9df18d5b8eaebc2c0e79a51e7c183b8707fbaf3a99d1dbda478a7cc57"} err="failed to get container status \"701a35a9df18d5b8eaebc2c0e79a51e7c183b8707fbaf3a99d1dbda478a7cc57\": rpc error: code = NotFound desc = could not find container \"701a35a9df18d5b8eaebc2c0e79a51e7c183b8707fbaf3a99d1dbda478a7cc57\": container with ID starting with 701a35a9df18d5b8eaebc2c0e79a51e7c183b8707fbaf3a99d1dbda478a7cc57 not found: ID does not exist" Oct 29 23:04:01.715882 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:01.715853 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs"] Oct 29 23:04:01.729487 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:01.729454 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0266dd-746db864c6-bhdrs"] Oct 29 23:04:02.175382 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:02.175345 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c7c4b21-7397-4db3-b170-aaded0958eaa" path="/var/lib/kubelet/pods/3c7c4b21-7397-4db3-b170-aaded0958eaa/volumes" Oct 29 23:04:03.123137 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:03.123103 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd"] Oct 29 23:04:03.123545 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:03.123386 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" podUID="5bfcb45f-0e94-4e58-bc91-9157be0569ed" containerName="istio-proxy" containerID="cri-o://414aca8c8c390f2ed7e4acea6e90f1ed4aecd0f4fc1a0b7b18fd5d1e87ff715e" gracePeriod=30 Oct 29 23:04:04.072367 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.072335 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg"] Oct 29 23:04:04.072968 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.072955 4821 cpu_manager.go:414] "containerMap: removing container" podUID="59c2911d-7186-44a7-9b7b-e5c55f1c185a" containerName="istio-proxy" Oct 29 23:04:04.073016 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.072971 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59c2911d-7186-44a7-9b7b-e5c55f1c185a" containerName="istio-proxy" Oct 29 23:04:04.073016 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.072981 4821 cpu_manager.go:414] "containerMap: removing container" podUID="3c7c4b21-7397-4db3-b170-aaded0958eaa" containerName="payload-processing" Oct 29 23:04:04.073016 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.072986 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c7c4b21-7397-4db3-b170-aaded0958eaa" containerName="payload-processing" Oct 29 23:04:04.073108 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.073018 4821 cpu_manager.go:414] "containerMap: removing container" podUID="15a298b4-75d8-4ca0-bad5-585c1e5c2058" containerName="payload-pre-processing" Oct 29 23:04:04.073108 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.073023 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15a298b4-75d8-4ca0-bad5-585c1e5c2058" containerName="payload-pre-processing" Oct 29 23:04:04.073108 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.073036 4821 cpu_manager.go:414] "containerMap: removing container" podUID="b1d1479f-84c6-46d1-b27c-65fa46cb9fcc" containerName="revoke-keys" Oct 29 23:04:04.073108 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.073041 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1d1479f-84c6-46d1-b27c-65fa46cb9fcc" containerName="revoke-keys" Oct 29 23:04:04.073108 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.073054 4821 cpu_manager.go:414] "containerMap: removing container" podUID="2d561e85-cff7-461f-9504-7edcac13fb00" containerName="payload-processing" Oct 29 23:04:04.073108 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.073059 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d561e85-cff7-461f-9504-7edcac13fb00" containerName="payload-processing" Oct 29 23:04:04.073309 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.073234 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="59c2911d-7186-44a7-9b7b-e5c55f1c185a" containerName="istio-proxy" Oct 29 23:04:04.073309 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.073274 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c7c4b21-7397-4db3-b170-aaded0958eaa" containerName="payload-processing" Oct 29 23:04:04.073309 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.073292 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="59c2911d-7186-44a7-9b7b-e5c55f1c185a" containerName="istio-proxy" Oct 29 23:04:04.073409 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.073310 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="15a298b4-75d8-4ca0-bad5-585c1e5c2058" containerName="payload-pre-processing" Oct 29 23:04:04.073409 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.073321 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1d1479f-84c6-46d1-b27c-65fa46cb9fcc" containerName="revoke-keys" Oct 29 23:04:04.073409 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.073332 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d561e85-cff7-461f-9504-7edcac13fb00" containerName="payload-processing" Oct 29 23:04:04.078533 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.078501 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg" Oct 29 23:04:04.098776 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.098747 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg"] Oct 29 23:04:04.105697 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.105608 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:04:04.105697 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.105671 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:04:04.239225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.224354 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w56f\" (UniqueName: \"kubernetes.io/projected/aec37f6d-fd80-4f7c-99bd-6029d1d71471-kube-api-access-9w56f\") pod \"maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg\" (UID: \"aec37f6d-fd80-4f7c-99bd-6029d1d71471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg" Oct 29 23:04:04.256448 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.256408 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aec37f6d-fd80-4f7c-99bd-6029d1d71471-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg\" (UID: \"aec37f6d-fd80-4f7c-99bd-6029d1d71471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg" Oct 29 23:04:04.256730 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.256707 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aec37f6d-fd80-4f7c-99bd-6029d1d71471-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg\" (UID: \"aec37f6d-fd80-4f7c-99bd-6029d1d71471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg" Oct 29 23:04:04.358522 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.358490 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w56f\" (UniqueName: \"kubernetes.io/projected/aec37f6d-fd80-4f7c-99bd-6029d1d71471-kube-api-access-9w56f\") pod \"maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg\" (UID: \"aec37f6d-fd80-4f7c-99bd-6029d1d71471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg" Oct 29 23:04:04.358733 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.358530 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aec37f6d-fd80-4f7c-99bd-6029d1d71471-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg\" (UID: \"aec37f6d-fd80-4f7c-99bd-6029d1d71471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg" Oct 29 23:04:04.358733 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.358581 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aec37f6d-fd80-4f7c-99bd-6029d1d71471-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg\" (UID: \"aec37f6d-fd80-4f7c-99bd-6029d1d71471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg" Oct 29 23:04:04.359312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.359289 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aec37f6d-fd80-4f7c-99bd-6029d1d71471-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg\" (UID: \"aec37f6d-fd80-4f7c-99bd-6029d1d71471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg" Oct 29 23:04:04.361216 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.361179 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aec37f6d-fd80-4f7c-99bd-6029d1d71471-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg\" (UID: \"aec37f6d-fd80-4f7c-99bd-6029d1d71471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg" Oct 29 23:04:04.383378 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.383347 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w56f\" (UniqueName: \"kubernetes.io/projected/aec37f6d-fd80-4f7c-99bd-6029d1d71471-kube-api-access-9w56f\") pod \"maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg\" (UID: \"aec37f6d-fd80-4f7c-99bd-6029d1d71471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg" Oct 29 23:04:04.397481 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.397456 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg" Oct 29 23:04:04.715673 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.715645 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg"] Oct 29 23:04:04.716112 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:04:04.716086 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec37f6d_fd80_4f7c_99bd_6029d1d71471.slice/crio-105c3642f0f39abf1b72696ae6f644d737e4cdab38a240ddf9a9fe307a9f6111 WatchSource:0}: Error finding container 105c3642f0f39abf1b72696ae6f644d737e4cdab38a240ddf9a9fe307a9f6111: Status 404 returned error can't find the container with id 105c3642f0f39abf1b72696ae6f644d737e4cdab38a240ddf9a9fe307a9f6111 Oct 29 23:04:04.735984 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.735954 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4"] Oct 29 23:04:04.744326 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:04.744295 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0266dd-lq5f4"] Oct 29 23:04:05.704600 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:05.704564 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg" event={"ID":"aec37f6d-fd80-4f7c-99bd-6029d1d71471","Type":"ContainerStarted","Data":"916359ebc8901531efd95c4bf5e188cfd3387f86674a4fb1782faad55576b1ef"} Oct 29 23:04:05.705004 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:05.704627 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg" event={"ID":"aec37f6d-fd80-4f7c-99bd-6029d1d71471","Type":"ContainerStarted","Data":"105c3642f0f39abf1b72696ae6f644d737e4cdab38a240ddf9a9fe307a9f6111"} Oct 29 23:04:05.747743 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:05.747689 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg" podStartSLOduration=1.747669348 podStartE2EDuration="1.747669348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:05.74664415 +0000 UTC m=+1299.687192060" watchObservedRunningTime="2026-10-29 23:04:05.747669348 +0000 UTC m=+1299.688217259" Oct 29 23:04:06.170926 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:06.170892 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1d1479f-84c6-46d1-b27c-65fa46cb9fcc" path="/var/lib/kubelet/pods/b1d1479f-84c6-46d1-b27c-65fa46cb9fcc/volumes" Oct 29 23:04:06.713194 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:06.713161 4821 generic.go:356] "Generic (PLEG): container finished" podID="aec37f6d-fd80-4f7c-99bd-6029d1d71471" containerID="916359ebc8901531efd95c4bf5e188cfd3387f86674a4fb1782faad55576b1ef" exitCode=0 Oct 29 23:04:06.713639 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:06.713264 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg" event={"ID":"aec37f6d-fd80-4f7c-99bd-6029d1d71471","Type":"ContainerDied","Data":"916359ebc8901531efd95c4bf5e188cfd3387f86674a4fb1782faad55576b1ef"} Oct 29 23:04:06.867659 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:06.867623 4821 patch_prober.go:29] interesting pod/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:04:06.867847 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:06.867682 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" podUID="5bfcb45f-0e94-4e58-bc91-9157be0569ed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:04:07.894778 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:07.894755 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg" Oct 29 23:04:08.020042 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.019963 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec37f6d-fd80-4f7c-99bd-6029d1d71471-kube-api-access-9w56f\" (UniqueName: \"kubernetes.io/projected/aec37f6d-fd80-4f7c-99bd-6029d1d71471-kube-api-access-9w56f\") pod \"aec37f6d-fd80-4f7c-99bd-6029d1d71471\" (UID: \"aec37f6d-fd80-4f7c-99bd-6029d1d71471\") " Oct 29 23:04:08.020244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.020192 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aec37f6d-fd80-4f7c-99bd-6029d1d71471-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aec37f6d-fd80-4f7c-99bd-6029d1d71471-maas-api-xks-ca\") pod \"aec37f6d-fd80-4f7c-99bd-6029d1d71471\" (UID: \"aec37f6d-fd80-4f7c-99bd-6029d1d71471\") " Oct 29 23:04:08.020325 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.020313 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aec37f6d-fd80-4f7c-99bd-6029d1d71471-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aec37f6d-fd80-4f7c-99bd-6029d1d71471-maas-api-openshift-ca\") pod \"aec37f6d-fd80-4f7c-99bd-6029d1d71471\" (UID: \"aec37f6d-fd80-4f7c-99bd-6029d1d71471\") " Oct 29 23:04:08.020658 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.020631 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec37f6d-fd80-4f7c-99bd-6029d1d71471-maas-api-openshift-ca" pod "aec37f6d-fd80-4f7c-99bd-6029d1d71471" (UID: "aec37f6d-fd80-4f7c-99bd-6029d1d71471"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:08.020926 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.020907 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aec37f6d-fd80-4f7c-99bd-6029d1d71471-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:08.022313 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.022281 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec37f6d-fd80-4f7c-99bd-6029d1d71471-kube-api-access-9w56f" pod "aec37f6d-fd80-4f7c-99bd-6029d1d71471" (UID: "aec37f6d-fd80-4f7c-99bd-6029d1d71471"). InnerVolumeSpecName "kube-api-access-9w56f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:08.022396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.022313 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aec37f6d-fd80-4f7c-99bd-6029d1d71471-maas-api-xks-ca" pod "aec37f6d-fd80-4f7c-99bd-6029d1d71471" (UID: "aec37f6d-fd80-4f7c-99bd-6029d1d71471"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:08.122928 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.122892 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aec37f6d-fd80-4f7c-99bd-6029d1d71471-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:08.122928 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.122922 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w56f\" (UniqueName: \"kubernetes.io/projected/aec37f6d-fd80-4f7c-99bd-6029d1d71471-kube-api-access-9w56f\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:08.403023 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.402989 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:04:08.560309 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.532121 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-envoy\") pod \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " Oct 29 23:04:08.560309 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.532255 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bfcb45f-0e94-4e58-bc91-9157be0569ed-kube-api-access-wmlff\" (UniqueName: \"kubernetes.io/projected/5bfcb45f-0e94-4e58-bc91-9157be0569ed-kube-api-access-wmlff\") pod \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " Oct 29 23:04:08.560309 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.532309 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-podinfo\") pod \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " Oct 29 23:04:08.560309 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.532346 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-token\" (UniqueName: \"kubernetes.io/projected/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-token\") pod \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " Oct 29 23:04:08.560309 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.532490 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-data\") pod \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " Oct 29 23:04:08.560309 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.532533 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-workload-certs\") pod \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " Oct 29 23:04:08.560309 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.532572 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-workload-socket\") pod \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " Oct 29 23:04:08.560309 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.532680 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istiod-ca-cert\") pod \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " Oct 29 23:04:08.560309 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.532771 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-credential-socket\") pod \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\" (UID: \"5bfcb45f-0e94-4e58-bc91-9157be0569ed\") " Oct 29 23:04:08.560309 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.553349 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-credential-socket" pod "5bfcb45f-0e94-4e58-bc91-9157be0569ed" (UID: "5bfcb45f-0e94-4e58-bc91-9157be0569ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:08.560309 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.559001 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-data" pod "5bfcb45f-0e94-4e58-bc91-9157be0569ed" (UID: "5bfcb45f-0e94-4e58-bc91-9157be0569ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:08.565038 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.564736 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfcb45f-0e94-4e58-bc91-9157be0569ed-kube-api-access-wmlff" pod "5bfcb45f-0e94-4e58-bc91-9157be0569ed" (UID: "5bfcb45f-0e94-4e58-bc91-9157be0569ed"). InnerVolumeSpecName "kube-api-access-wmlff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:08.565845 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.565799 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-workload-socket" pod "5bfcb45f-0e94-4e58-bc91-9157be0569ed" (UID: "5bfcb45f-0e94-4e58-bc91-9157be0569ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:08.566572 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.566153 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-workload-certs" pod "5bfcb45f-0e94-4e58-bc91-9157be0569ed" (UID: "5bfcb45f-0e94-4e58-bc91-9157be0569ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:08.566572 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.566497 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istiod-ca-cert" pod "5bfcb45f-0e94-4e58-bc91-9157be0569ed" (UID: "5bfcb45f-0e94-4e58-bc91-9157be0569ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:08.580820 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.578250 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-envoy" pod "5bfcb45f-0e94-4e58-bc91-9157be0569ed" (UID: "5bfcb45f-0e94-4e58-bc91-9157be0569ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:08.581253 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.581129 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-podinfo" pod "5bfcb45f-0e94-4e58-bc91-9157be0569ed" (UID: "5bfcb45f-0e94-4e58-bc91-9157be0569ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:08.639419 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.639187 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:08.639419 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.639251 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:08.639419 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.639265 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:08.639419 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.639280 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmlff\" (UniqueName: \"kubernetes.io/projected/5bfcb45f-0e94-4e58-bc91-9157be0569ed-kube-api-access-wmlff\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:08.639419 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.639294 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:08.639419 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.639309 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:08.639419 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.639324 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:08.639419 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.639336 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfcb45f-0e94-4e58-bc91-9157be0569ed-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:08.727383 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.722490 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-token" pod "5bfcb45f-0e94-4e58-bc91-9157be0569ed" (UID: "5bfcb45f-0e94-4e58-bc91-9157be0569ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:08.741279 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.741099 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfcb45f-0e94-4e58-bc91-9157be0569ed-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:08.747982 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.747955 4821 generic.go:356] "Generic (PLEG): container finished" podID="5bfcb45f-0e94-4e58-bc91-9157be0569ed" containerID="414aca8c8c390f2ed7e4acea6e90f1ed4aecd0f4fc1a0b7b18fd5d1e87ff715e" exitCode=0 Oct 29 23:04:08.748144 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.748007 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" Oct 29 23:04:08.748144 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.748039 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" event={"ID":"5bfcb45f-0e94-4e58-bc91-9157be0569ed","Type":"ContainerDied","Data":"414aca8c8c390f2ed7e4acea6e90f1ed4aecd0f4fc1a0b7b18fd5d1e87ff715e"} Oct 29 23:04:08.748144 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.748107 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd" event={"ID":"5bfcb45f-0e94-4e58-bc91-9157be0569ed","Type":"ContainerDied","Data":"abcf65a14f426b1562ccdb59be9c63e45f7bb722fa4200415761190dccf3891f"} Oct 29 23:04:08.748144 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.748140 4821 scope.go:122] "RemoveContainer" containerID="414aca8c8c390f2ed7e4acea6e90f1ed4aecd0f4fc1a0b7b18fd5d1e87ff715e" Oct 29 23:04:08.751610 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.750437 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg" event={"ID":"aec37f6d-fd80-4f7c-99bd-6029d1d71471","Type":"ContainerDied","Data":"105c3642f0f39abf1b72696ae6f644d737e4cdab38a240ddf9a9fe307a9f6111"} Oct 29 23:04:08.751610 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.750480 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="105c3642f0f39abf1b72696ae6f644d737e4cdab38a240ddf9a9fe307a9f6111" Oct 29 23:04:08.751610 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.750558 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg" Oct 29 23:04:08.761304 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.761285 4821 scope.go:122] "RemoveContainer" containerID="c7c4d2d9ebfd51c6aeb68c9d6aebaaccffae8ac0057e67f724ef1bb81fd13a00" Oct 29 23:04:08.774295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.774273 4821 scope.go:122] "RemoveContainer" containerID="414aca8c8c390f2ed7e4acea6e90f1ed4aecd0f4fc1a0b7b18fd5d1e87ff715e" Oct 29 23:04:08.774584 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:04:08.774556 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"414aca8c8c390f2ed7e4acea6e90f1ed4aecd0f4fc1a0b7b18fd5d1e87ff715e\": container with ID starting with 414aca8c8c390f2ed7e4acea6e90f1ed4aecd0f4fc1a0b7b18fd5d1e87ff715e not found: ID does not exist" containerID="414aca8c8c390f2ed7e4acea6e90f1ed4aecd0f4fc1a0b7b18fd5d1e87ff715e" Oct 29 23:04:08.774656 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.774596 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"414aca8c8c390f2ed7e4acea6e90f1ed4aecd0f4fc1a0b7b18fd5d1e87ff715e"} err="failed to get container status \"414aca8c8c390f2ed7e4acea6e90f1ed4aecd0f4fc1a0b7b18fd5d1e87ff715e\": rpc error: code = NotFound desc = could not find container \"414aca8c8c390f2ed7e4acea6e90f1ed4aecd0f4fc1a0b7b18fd5d1e87ff715e\": container with ID starting with 414aca8c8c390f2ed7e4acea6e90f1ed4aecd0f4fc1a0b7b18fd5d1e87ff715e not found: ID does not exist" Oct 29 23:04:08.774656 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.774620 4821 scope.go:122] "RemoveContainer" containerID="c7c4d2d9ebfd51c6aeb68c9d6aebaaccffae8ac0057e67f724ef1bb81fd13a00" Oct 29 23:04:08.774897 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:04:08.774870 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c4d2d9ebfd51c6aeb68c9d6aebaaccffae8ac0057e67f724ef1bb81fd13a00\": container with ID starting with c7c4d2d9ebfd51c6aeb68c9d6aebaaccffae8ac0057e67f724ef1bb81fd13a00 not found: ID does not exist" containerID="c7c4d2d9ebfd51c6aeb68c9d6aebaaccffae8ac0057e67f724ef1bb81fd13a00" Oct 29 23:04:08.774945 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.774902 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c4d2d9ebfd51c6aeb68c9d6aebaaccffae8ac0057e67f724ef1bb81fd13a00"} err="failed to get container status \"c7c4d2d9ebfd51c6aeb68c9d6aebaaccffae8ac0057e67f724ef1bb81fd13a00\": rpc error: code = NotFound desc = could not find container \"c7c4d2d9ebfd51c6aeb68c9d6aebaaccffae8ac0057e67f724ef1bb81fd13a00\": container with ID starting with c7c4d2d9ebfd51c6aeb68c9d6aebaaccffae8ac0057e67f724ef1bb81fd13a00 not found: ID does not exist" Oct 29 23:04:08.806768 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.806734 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd"] Oct 29 23:04:08.811122 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:08.811060 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f173b1-openshift-default-7457cf4fbc-bpbvd"] Oct 29 23:04:10.171986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:10.171954 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bfcb45f-0e94-4e58-bc91-9157be0569ed" path="/var/lib/kubelet/pods/5bfcb45f-0e94-4e58-bc91-9157be0569ed/volumes" Oct 29 23:04:12.129462 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.129426 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5"] Oct 29 23:04:12.129924 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.129655 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" podUID="e778eb53-b230-451e-b7ae-3d8512366c59" containerName="istio-proxy" containerID="cri-o://6720048b4ee28892faf8bbec09bddd8c5caa536c4959d46171a078a6908fac02" gracePeriod=30 Oct 29 23:04:12.139706 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.139677 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2"] Oct 29 23:04:12.139932 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.139904 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" podUID="dcc509cf-1d6b-4453-b46b-d756cd4ec293" containerName="istio-proxy" containerID="cri-o://6082b264f6820cb7d0dcf63f2b1d01654d05fafd24bbb5e11ddadf89ca45f503" gracePeriod=30 Oct 29 23:04:12.145066 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.145040 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552"] Oct 29 23:04:12.145259 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.145238 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" podUID="39efdaf8-97f8-47fc-964c-4bc0daab505f" containerName="istio-proxy" containerID="cri-o://0fe29c8f2b6d2d9654843ba173b17de1a3b6508e9a4dd2242b6b420d8e026557" gracePeriod=30 Oct 29 23:04:12.681376 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.681348 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr"] Oct 29 23:04:12.681938 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.681926 4821 cpu_manager.go:414] "containerMap: removing container" podUID="5bfcb45f-0e94-4e58-bc91-9157be0569ed" containerName="istio-proxy" Oct 29 23:04:12.681992 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.681940 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bfcb45f-0e94-4e58-bc91-9157be0569ed" containerName="istio-proxy" Oct 29 23:04:12.681992 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.681963 4821 cpu_manager.go:414] "containerMap: removing container" podUID="aec37f6d-fd80-4f7c-99bd-6029d1d71471" containerName="revoke-keys" Oct 29 23:04:12.681992 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.681968 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec37f6d-fd80-4f7c-99bd-6029d1d71471" containerName="revoke-keys" Oct 29 23:04:12.681992 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.681980 4821 cpu_manager.go:414] "containerMap: removing container" podUID="59c2911d-7186-44a7-9b7b-e5c55f1c185a" containerName="istio-proxy" Oct 29 23:04:12.681992 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.681986 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59c2911d-7186-44a7-9b7b-e5c55f1c185a" containerName="istio-proxy" Oct 29 23:04:12.682139 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.682086 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec37f6d-fd80-4f7c-99bd-6029d1d71471" containerName="revoke-keys" Oct 29 23:04:12.682139 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.682098 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bfcb45f-0e94-4e58-bc91-9157be0569ed" containerName="istio-proxy" Oct 29 23:04:12.682139 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.682109 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bfcb45f-0e94-4e58-bc91-9157be0569ed" containerName="istio-proxy" Oct 29 23:04:12.687283 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.687260 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr" Oct 29 23:04:12.705227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.705180 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:04:12.705353 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.705227 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:04:12.707101 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.707075 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr"] Oct 29 23:04:12.849755 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.849722 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8wpz\" (UniqueName: \"kubernetes.io/projected/c582c536-c024-4bb5-a50e-8d78ca55f55f-kube-api-access-c8wpz\") pod \"maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr\" (UID: \"c582c536-c024-4bb5-a50e-8d78ca55f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr" Oct 29 23:04:12.849909 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.849793 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c582c536-c024-4bb5-a50e-8d78ca55f55f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr\" (UID: \"c582c536-c024-4bb5-a50e-8d78ca55f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr" Oct 29 23:04:12.849976 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.849928 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c582c536-c024-4bb5-a50e-8d78ca55f55f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr\" (UID: \"c582c536-c024-4bb5-a50e-8d78ca55f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr" Oct 29 23:04:12.951192 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.951099 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8wpz\" (UniqueName: \"kubernetes.io/projected/c582c536-c024-4bb5-a50e-8d78ca55f55f-kube-api-access-c8wpz\") pod \"maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr\" (UID: \"c582c536-c024-4bb5-a50e-8d78ca55f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr" Oct 29 23:04:12.951412 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.951187 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c582c536-c024-4bb5-a50e-8d78ca55f55f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr\" (UID: \"c582c536-c024-4bb5-a50e-8d78ca55f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr" Oct 29 23:04:12.951412 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.951291 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c582c536-c024-4bb5-a50e-8d78ca55f55f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr\" (UID: \"c582c536-c024-4bb5-a50e-8d78ca55f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr" Oct 29 23:04:12.952052 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.951962 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c582c536-c024-4bb5-a50e-8d78ca55f55f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr\" (UID: \"c582c536-c024-4bb5-a50e-8d78ca55f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr" Oct 29 23:04:12.953711 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.953688 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c582c536-c024-4bb5-a50e-8d78ca55f55f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr\" (UID: \"c582c536-c024-4bb5-a50e-8d78ca55f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr" Oct 29 23:04:12.975540 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:12.975520 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8wpz\" (UniqueName: \"kubernetes.io/projected/c582c536-c024-4bb5-a50e-8d78ca55f55f-kube-api-access-c8wpz\") pod \"maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr\" (UID: \"c582c536-c024-4bb5-a50e-8d78ca55f55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr" Oct 29 23:04:13.001605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:13.001574 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr" Oct 29 23:04:13.312010 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:13.311981 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr"] Oct 29 23:04:13.312355 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:04:13.312027 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc582c536_c024_4bb5_a50e_8d78ca55f55f.slice/crio-5b3f795386b066a8cc9ba86b70af72c5119bb019fb9d10709129f7d960eb99d4 WatchSource:0}: Error finding container 5b3f795386b066a8cc9ba86b70af72c5119bb019fb9d10709129f7d960eb99d4: Status 404 returned error can't find the container with id 5b3f795386b066a8cc9ba86b70af72c5119bb019fb9d10709129f7d960eb99d4 Oct 29 23:04:13.832666 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:13.832632 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr" event={"ID":"c582c536-c024-4bb5-a50e-8d78ca55f55f","Type":"ContainerStarted","Data":"d9374905b10875f93d83af569ebe875f6b7fc25d388b036f2b94487ec018f618"} Oct 29 23:04:13.832848 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:13.832687 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr" event={"ID":"c582c536-c024-4bb5-a50e-8d78ca55f55f","Type":"ContainerStarted","Data":"5b3f795386b066a8cc9ba86b70af72c5119bb019fb9d10709129f7d960eb99d4"} Oct 29 23:04:13.875338 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:13.875280 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr" podStartSLOduration=1.875266779 podStartE2EDuration="1.875266779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:13.874087804 +0000 UTC m=+1307.814635713" watchObservedRunningTime="2026-10-29 23:04:13.875266779 +0000 UTC m=+1307.815814690" Oct 29 23:04:14.843763 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:14.843726 4821 generic.go:356] "Generic (PLEG): container finished" podID="c582c536-c024-4bb5-a50e-8d78ca55f55f" containerID="d9374905b10875f93d83af569ebe875f6b7fc25d388b036f2b94487ec018f618" exitCode=0 Oct 29 23:04:14.844171 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:14.843818 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr" event={"ID":"c582c536-c024-4bb5-a50e-8d78ca55f55f","Type":"ContainerDied","Data":"d9374905b10875f93d83af569ebe875f6b7fc25d388b036f2b94487ec018f618"} Oct 29 23:04:16.063464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:16.063432 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr" Oct 29 23:04:16.203006 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:16.202970 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c582c536-c024-4bb5-a50e-8d78ca55f55f-kube-api-access-c8wpz\" (UniqueName: \"kubernetes.io/projected/c582c536-c024-4bb5-a50e-8d78ca55f55f-kube-api-access-c8wpz\") pod \"c582c536-c024-4bb5-a50e-8d78ca55f55f\" (UID: \"c582c536-c024-4bb5-a50e-8d78ca55f55f\") " Oct 29 23:04:16.203161 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:16.203074 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c582c536-c024-4bb5-a50e-8d78ca55f55f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c582c536-c024-4bb5-a50e-8d78ca55f55f-maas-api-openshift-ca\") pod \"c582c536-c024-4bb5-a50e-8d78ca55f55f\" (UID: \"c582c536-c024-4bb5-a50e-8d78ca55f55f\") " Oct 29 23:04:16.203412 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:16.203295 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c582c536-c024-4bb5-a50e-8d78ca55f55f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c582c536-c024-4bb5-a50e-8d78ca55f55f-maas-api-xks-ca\") pod \"c582c536-c024-4bb5-a50e-8d78ca55f55f\" (UID: \"c582c536-c024-4bb5-a50e-8d78ca55f55f\") " Oct 29 23:04:16.203716 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:16.203690 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c582c536-c024-4bb5-a50e-8d78ca55f55f-maas-api-openshift-ca" pod "c582c536-c024-4bb5-a50e-8d78ca55f55f" (UID: "c582c536-c024-4bb5-a50e-8d78ca55f55f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:16.204025 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:16.204005 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c582c536-c024-4bb5-a50e-8d78ca55f55f-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:16.206224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:16.206182 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c582c536-c024-4bb5-a50e-8d78ca55f55f-maas-api-xks-ca" pod "c582c536-c024-4bb5-a50e-8d78ca55f55f" (UID: "c582c536-c024-4bb5-a50e-8d78ca55f55f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:16.206409 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:16.206243 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c582c536-c024-4bb5-a50e-8d78ca55f55f-kube-api-access-c8wpz" pod "c582c536-c024-4bb5-a50e-8d78ca55f55f" (UID: "c582c536-c024-4bb5-a50e-8d78ca55f55f"). InnerVolumeSpecName "kube-api-access-c8wpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:16.306286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:16.306249 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c582c536-c024-4bb5-a50e-8d78ca55f55f-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:16.306286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:16.306283 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8wpz\" (UniqueName: \"kubernetes.io/projected/c582c536-c024-4bb5-a50e-8d78ca55f55f-kube-api-access-c8wpz\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:16.860951 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:16.860918 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr" Oct 29 23:04:16.860951 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:16.860939 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr" event={"ID":"c582c536-c024-4bb5-a50e-8d78ca55f55f","Type":"ContainerDied","Data":"5b3f795386b066a8cc9ba86b70af72c5119bb019fb9d10709129f7d960eb99d4"} Oct 29 23:04:16.861250 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:16.860982 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b3f795386b066a8cc9ba86b70af72c5119bb019fb9d10709129f7d960eb99d4" Oct 29 23:04:17.495006 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.494968 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:04:17.501042 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.501014 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:04:17.525280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.525251 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-credential-socket\") pod \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " Oct 29 23:04:17.525436 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.525399 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc509cf-1d6b-4453-b46b-d756cd4ec293-kube-api-access-4g4k4\" (UniqueName: \"kubernetes.io/projected/dcc509cf-1d6b-4453-b46b-d756cd4ec293-kube-api-access-4g4k4\") pod \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " Oct 29 23:04:17.525504 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.525443 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-istio-envoy\") pod \"e778eb53-b230-451e-b7ae-3d8512366c59\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " Oct 29 23:04:17.525504 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.525471 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-credential-socket" pod "dcc509cf-1d6b-4453-b46b-d756cd4ec293" (UID: "dcc509cf-1d6b-4453-b46b-d756cd4ec293"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.525627 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.525502 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-workload-certs\") pod \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " Oct 29 23:04:17.525627 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.525525 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-credential-socket\") pod \"e778eb53-b230-451e-b7ae-3d8512366c59\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " Oct 29 23:04:17.525627 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.525571 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e778eb53-b230-451e-b7ae-3d8512366c59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e778eb53-b230-451e-b7ae-3d8512366c59-istio-podinfo\") pod \"e778eb53-b230-451e-b7ae-3d8512366c59\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " Oct 29 23:04:17.525627 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.525606 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-podinfo\") pod \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " Oct 29 23:04:17.525840 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.525656 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-data\") pod \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " Oct 29 23:04:17.525840 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.525689 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e778eb53-b230-451e-b7ae-3d8512366c59-kube-api-access-q8ls9\" (UniqueName: \"kubernetes.io/projected/e778eb53-b230-451e-b7ae-3d8512366c59-kube-api-access-q8ls9\") pod \"e778eb53-b230-451e-b7ae-3d8512366c59\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " Oct 29 23:04:17.525840 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.525776 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istiod-ca-cert\") pod \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " Oct 29 23:04:17.525840 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.525808 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e778eb53-b230-451e-b7ae-3d8512366c59-istio-token\" (UniqueName: \"kubernetes.io/projected/e778eb53-b230-451e-b7ae-3d8512366c59-istio-token\") pod \"e778eb53-b230-451e-b7ae-3d8512366c59\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " Oct 29 23:04:17.525960 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.525867 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-token\" (UniqueName: \"kubernetes.io/projected/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-token\") pod \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " Oct 29 23:04:17.526000 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.525959 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-workload-certs\") pod \"e778eb53-b230-451e-b7ae-3d8512366c59\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " Oct 29 23:04:17.526000 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.525987 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-istio-data\") pod \"e778eb53-b230-451e-b7ae-3d8512366c59\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " Oct 29 23:04:17.526065 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.526022 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e778eb53-b230-451e-b7ae-3d8512366c59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e778eb53-b230-451e-b7ae-3d8512366c59-istiod-ca-cert\") pod \"e778eb53-b230-451e-b7ae-3d8512366c59\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " Oct 29 23:04:17.526098 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.526080 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-workload-socket\") pod \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " Oct 29 23:04:17.526140 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.526118 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-envoy\") pod \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\" (UID: \"dcc509cf-1d6b-4453-b46b-d756cd4ec293\") " Oct 29 23:04:17.526789 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.526306 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-workload-socket\") pod \"e778eb53-b230-451e-b7ae-3d8512366c59\" (UID: \"e778eb53-b230-451e-b7ae-3d8512366c59\") " Oct 29 23:04:17.526789 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.526544 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-workload-certs" pod "dcc509cf-1d6b-4453-b46b-d756cd4ec293" (UID: "dcc509cf-1d6b-4453-b46b-d756cd4ec293"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.526789 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.526689 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istiod-ca-cert" pod "dcc509cf-1d6b-4453-b46b-d756cd4ec293" (UID: "dcc509cf-1d6b-4453-b46b-d756cd4ec293"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:17.526789 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.526691 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-credential-socket" pod "e778eb53-b230-451e-b7ae-3d8512366c59" (UID: "e778eb53-b230-451e-b7ae-3d8512366c59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.527516 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.527495 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.527592 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.527523 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.527592 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.527539 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.527592 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.527553 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.528530 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.528491 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-data" pod "dcc509cf-1d6b-4453-b46b-d756cd4ec293" (UID: "dcc509cf-1d6b-4453-b46b-d756cd4ec293"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.528530 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.528518 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-istio-data" pod "e778eb53-b230-451e-b7ae-3d8512366c59" (UID: "e778eb53-b230-451e-b7ae-3d8512366c59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.528747 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.528682 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-workload-certs" pod "e778eb53-b230-451e-b7ae-3d8512366c59" (UID: "e778eb53-b230-451e-b7ae-3d8512366c59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.528877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.528850 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-workload-socket" pod "dcc509cf-1d6b-4453-b46b-d756cd4ec293" (UID: "dcc509cf-1d6b-4453-b46b-d756cd4ec293"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.528942 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.528866 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:04:17.529016 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.528992 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc509cf-1d6b-4453-b46b-d756cd4ec293-kube-api-access-4g4k4" pod "dcc509cf-1d6b-4453-b46b-d756cd4ec293" (UID: "dcc509cf-1d6b-4453-b46b-d756cd4ec293"). InnerVolumeSpecName "kube-api-access-4g4k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:17.529547 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.529511 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e778eb53-b230-451e-b7ae-3d8512366c59-istiod-ca-cert" pod "e778eb53-b230-451e-b7ae-3d8512366c59" (UID: "e778eb53-b230-451e-b7ae-3d8512366c59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:17.529996 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.529974 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-workload-socket" pod "e778eb53-b230-451e-b7ae-3d8512366c59" (UID: "e778eb53-b230-451e-b7ae-3d8512366c59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.530316 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.530295 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-istio-envoy" pod "e778eb53-b230-451e-b7ae-3d8512366c59" (UID: "e778eb53-b230-451e-b7ae-3d8512366c59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.531691 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.531575 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-token" pod "dcc509cf-1d6b-4453-b46b-d756cd4ec293" (UID: "dcc509cf-1d6b-4453-b46b-d756cd4ec293"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:17.531691 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.531663 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-envoy" pod "dcc509cf-1d6b-4453-b46b-d756cd4ec293" (UID: "dcc509cf-1d6b-4453-b46b-d756cd4ec293"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.532081 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.532058 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e778eb53-b230-451e-b7ae-3d8512366c59-istio-token" pod "e778eb53-b230-451e-b7ae-3d8512366c59" (UID: "e778eb53-b230-451e-b7ae-3d8512366c59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:17.532748 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.532721 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e778eb53-b230-451e-b7ae-3d8512366c59-istio-podinfo" pod "e778eb53-b230-451e-b7ae-3d8512366c59" (UID: "e778eb53-b230-451e-b7ae-3d8512366c59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:17.533004 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.532979 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-podinfo" pod "dcc509cf-1d6b-4453-b46b-d756cd4ec293" (UID: "dcc509cf-1d6b-4453-b46b-d756cd4ec293"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:17.533870 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.533852 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e778eb53-b230-451e-b7ae-3d8512366c59-kube-api-access-q8ls9" pod "e778eb53-b230-451e-b7ae-3d8512366c59" (UID: "e778eb53-b230-451e-b7ae-3d8512366c59"). InnerVolumeSpecName "kube-api-access-q8ls9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:17.628846 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.628813 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-token\" (UniqueName: \"kubernetes.io/projected/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-token\") pod \"39efdaf8-97f8-47fc-964c-4bc0daab505f\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " Oct 29 23:04:17.629037 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.628859 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39efdaf8-97f8-47fc-964c-4bc0daab505f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39efdaf8-97f8-47fc-964c-4bc0daab505f-istiod-ca-cert\") pod \"39efdaf8-97f8-47fc-964c-4bc0daab505f\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " Oct 29 23:04:17.629037 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.628904 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-envoy\") pod \"39efdaf8-97f8-47fc-964c-4bc0daab505f\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " Oct 29 23:04:17.629037 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.628935 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-podinfo\") pod \"39efdaf8-97f8-47fc-964c-4bc0daab505f\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " Oct 29 23:04:17.629037 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.629004 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-workload-socket\") pod \"39efdaf8-97f8-47fc-964c-4bc0daab505f\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " Oct 29 23:04:17.629037 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.629029 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-credential-socket\") pod \"39efdaf8-97f8-47fc-964c-4bc0daab505f\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " Oct 29 23:04:17.629331 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.629122 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39efdaf8-97f8-47fc-964c-4bc0daab505f-kube-api-access-bkgsj\" (UniqueName: \"kubernetes.io/projected/39efdaf8-97f8-47fc-964c-4bc0daab505f-kube-api-access-bkgsj\") pod \"39efdaf8-97f8-47fc-964c-4bc0daab505f\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " Oct 29 23:04:17.629331 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.629219 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-workload-certs\") pod \"39efdaf8-97f8-47fc-964c-4bc0daab505f\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " Oct 29 23:04:17.629977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.629472 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-data\") pod \"39efdaf8-97f8-47fc-964c-4bc0daab505f\" (UID: \"39efdaf8-97f8-47fc-964c-4bc0daab505f\") " Oct 29 23:04:17.629977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.629492 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-credential-socket" pod "39efdaf8-97f8-47fc-964c-4bc0daab505f" (UID: "39efdaf8-97f8-47fc-964c-4bc0daab505f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.629977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.629686 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-workload-certs" pod "39efdaf8-97f8-47fc-964c-4bc0daab505f" (UID: "39efdaf8-97f8-47fc-964c-4bc0daab505f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.629977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.629807 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-workload-socket" pod "39efdaf8-97f8-47fc-964c-4bc0daab505f" (UID: "39efdaf8-97f8-47fc-964c-4bc0daab505f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.629977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.629915 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39efdaf8-97f8-47fc-964c-4bc0daab505f-istiod-ca-cert" pod "39efdaf8-97f8-47fc-964c-4bc0daab505f" (UID: "39efdaf8-97f8-47fc-964c-4bc0daab505f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:17.630357 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.630080 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-data" pod "39efdaf8-97f8-47fc-964c-4bc0daab505f" (UID: "39efdaf8-97f8-47fc-964c-4bc0daab505f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.630413 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.630366 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.630413 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.630389 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.630413 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.630405 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39efdaf8-97f8-47fc-964c-4bc0daab505f-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.630621 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.630419 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g4k4\" (UniqueName: \"kubernetes.io/projected/dcc509cf-1d6b-4453-b46b-d756cd4ec293-kube-api-access-4g4k4\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.630621 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.630480 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.630621 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.630581 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.630621 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.630617 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.630823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.630630 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e778eb53-b230-451e-b7ae-3d8512366c59-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.630823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.630644 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.630823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.630657 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.630823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.630672 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8ls9\" (UniqueName: \"kubernetes.io/projected/e778eb53-b230-451e-b7ae-3d8512366c59-kube-api-access-q8ls9\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.630823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.630685 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e778eb53-b230-451e-b7ae-3d8512366c59-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.630823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.630698 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.630823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.630710 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.630823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.630723 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.630823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.630738 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e778eb53-b230-451e-b7ae-3d8512366c59-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.630823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.630751 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e778eb53-b230-451e-b7ae-3d8512366c59-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.630823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.630764 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.630823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.630781 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc509cf-1d6b-4453-b46b-d756cd4ec293-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.631719 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.631698 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-token" pod "39efdaf8-97f8-47fc-964c-4bc0daab505f" (UID: "39efdaf8-97f8-47fc-964c-4bc0daab505f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:17.632115 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.632097 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-envoy" pod "39efdaf8-97f8-47fc-964c-4bc0daab505f" (UID: "39efdaf8-97f8-47fc-964c-4bc0daab505f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:17.632548 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.632528 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-podinfo" pod "39efdaf8-97f8-47fc-964c-4bc0daab505f" (UID: "39efdaf8-97f8-47fc-964c-4bc0daab505f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:17.632938 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.632912 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39efdaf8-97f8-47fc-964c-4bc0daab505f-kube-api-access-bkgsj" pod "39efdaf8-97f8-47fc-964c-4bc0daab505f" (UID: "39efdaf8-97f8-47fc-964c-4bc0daab505f"). InnerVolumeSpecName "kube-api-access-bkgsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:17.732921 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.732839 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkgsj\" (UniqueName: \"kubernetes.io/projected/39efdaf8-97f8-47fc-964c-4bc0daab505f-kube-api-access-bkgsj\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.732921 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.732882 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.732921 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.732896 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.732921 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.732908 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39efdaf8-97f8-47fc-964c-4bc0daab505f-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:17.869852 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.869821 4821 generic.go:356] "Generic (PLEG): container finished" podID="39efdaf8-97f8-47fc-964c-4bc0daab505f" containerID="0fe29c8f2b6d2d9654843ba173b17de1a3b6508e9a4dd2242b6b420d8e026557" exitCode=0 Oct 29 23:04:17.870045 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.869871 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" Oct 29 23:04:17.870045 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.869897 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" event={"ID":"39efdaf8-97f8-47fc-964c-4bc0daab505f","Type":"ContainerDied","Data":"0fe29c8f2b6d2d9654843ba173b17de1a3b6508e9a4dd2242b6b420d8e026557"} Oct 29 23:04:17.870045 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.869956 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552" event={"ID":"39efdaf8-97f8-47fc-964c-4bc0daab505f","Type":"ContainerDied","Data":"af6f522d6cee952edab2b1a636f049859af2041f8c21ebc1956bfc2bbde9ad04"} Oct 29 23:04:17.870045 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.869984 4821 scope.go:122] "RemoveContainer" containerID="0fe29c8f2b6d2d9654843ba173b17de1a3b6508e9a4dd2242b6b420d8e026557" Oct 29 23:04:17.871620 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.871595 4821 generic.go:356] "Generic (PLEG): container finished" podID="dcc509cf-1d6b-4453-b46b-d756cd4ec293" containerID="6082b264f6820cb7d0dcf63f2b1d01654d05fafd24bbb5e11ddadf89ca45f503" exitCode=0 Oct 29 23:04:17.871741 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.871650 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" Oct 29 23:04:17.871741 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.871676 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" event={"ID":"dcc509cf-1d6b-4453-b46b-d756cd4ec293","Type":"ContainerDied","Data":"6082b264f6820cb7d0dcf63f2b1d01654d05fafd24bbb5e11ddadf89ca45f503"} Oct 29 23:04:17.871741 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.871736 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2" event={"ID":"dcc509cf-1d6b-4453-b46b-d756cd4ec293","Type":"ContainerDied","Data":"28155a18a4507ef7074323931ea8e1b740dcd3554536809c87f6ead7ec6889ff"} Oct 29 23:04:17.873920 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.873893 4821 generic.go:356] "Generic (PLEG): container finished" podID="e778eb53-b230-451e-b7ae-3d8512366c59" containerID="6720048b4ee28892faf8bbec09bddd8c5caa536c4959d46171a078a6908fac02" exitCode=0 Oct 29 23:04:17.874325 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.873999 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" event={"ID":"e778eb53-b230-451e-b7ae-3d8512366c59","Type":"ContainerDied","Data":"6720048b4ee28892faf8bbec09bddd8c5caa536c4959d46171a078a6908fac02"} Oct 29 23:04:17.874325 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.874042 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" event={"ID":"e778eb53-b230-451e-b7ae-3d8512366c59","Type":"ContainerDied","Data":"c7256a75f7f86cccc9d8a466bc0ccfe17cae2269e9afbdb37ff7bdeed8f741fb"} Oct 29 23:04:17.874325 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.874109 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5" Oct 29 23:04:17.884663 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.884643 4821 scope.go:122] "RemoveContainer" containerID="0fe29c8f2b6d2d9654843ba173b17de1a3b6508e9a4dd2242b6b420d8e026557" Oct 29 23:04:17.884992 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:04:17.884965 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fe29c8f2b6d2d9654843ba173b17de1a3b6508e9a4dd2242b6b420d8e026557\": container with ID starting with 0fe29c8f2b6d2d9654843ba173b17de1a3b6508e9a4dd2242b6b420d8e026557 not found: ID does not exist" containerID="0fe29c8f2b6d2d9654843ba173b17de1a3b6508e9a4dd2242b6b420d8e026557" Oct 29 23:04:17.885077 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.885005 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe29c8f2b6d2d9654843ba173b17de1a3b6508e9a4dd2242b6b420d8e026557"} err="failed to get container status \"0fe29c8f2b6d2d9654843ba173b17de1a3b6508e9a4dd2242b6b420d8e026557\": rpc error: code = NotFound desc = could not find container \"0fe29c8f2b6d2d9654843ba173b17de1a3b6508e9a4dd2242b6b420d8e026557\": container with ID starting with 0fe29c8f2b6d2d9654843ba173b17de1a3b6508e9a4dd2242b6b420d8e026557 not found: ID does not exist" Oct 29 23:04:17.885077 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.885028 4821 scope.go:122] "RemoveContainer" containerID="6082b264f6820cb7d0dcf63f2b1d01654d05fafd24bbb5e11ddadf89ca45f503" Oct 29 23:04:17.896946 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.896926 4821 scope.go:122] "RemoveContainer" containerID="6082b264f6820cb7d0dcf63f2b1d01654d05fafd24bbb5e11ddadf89ca45f503" Oct 29 23:04:17.897261 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:04:17.897239 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6082b264f6820cb7d0dcf63f2b1d01654d05fafd24bbb5e11ddadf89ca45f503\": container with ID starting with 6082b264f6820cb7d0dcf63f2b1d01654d05fafd24bbb5e11ddadf89ca45f503 not found: ID does not exist" containerID="6082b264f6820cb7d0dcf63f2b1d01654d05fafd24bbb5e11ddadf89ca45f503" Oct 29 23:04:17.897346 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.897275 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6082b264f6820cb7d0dcf63f2b1d01654d05fafd24bbb5e11ddadf89ca45f503"} err="failed to get container status \"6082b264f6820cb7d0dcf63f2b1d01654d05fafd24bbb5e11ddadf89ca45f503\": rpc error: code = NotFound desc = could not find container \"6082b264f6820cb7d0dcf63f2b1d01654d05fafd24bbb5e11ddadf89ca45f503\": container with ID starting with 6082b264f6820cb7d0dcf63f2b1d01654d05fafd24bbb5e11ddadf89ca45f503 not found: ID does not exist" Oct 29 23:04:17.897346 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.897297 4821 scope.go:122] "RemoveContainer" containerID="6720048b4ee28892faf8bbec09bddd8c5caa536c4959d46171a078a6908fac02" Oct 29 23:04:17.907344 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.907328 4821 scope.go:122] "RemoveContainer" containerID="6720048b4ee28892faf8bbec09bddd8c5caa536c4959d46171a078a6908fac02" Oct 29 23:04:17.907626 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:04:17.907606 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6720048b4ee28892faf8bbec09bddd8c5caa536c4959d46171a078a6908fac02\": container with ID starting with 6720048b4ee28892faf8bbec09bddd8c5caa536c4959d46171a078a6908fac02 not found: ID does not exist" containerID="6720048b4ee28892faf8bbec09bddd8c5caa536c4959d46171a078a6908fac02" Oct 29 23:04:17.907675 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.907634 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6720048b4ee28892faf8bbec09bddd8c5caa536c4959d46171a078a6908fac02"} err="failed to get container status \"6720048b4ee28892faf8bbec09bddd8c5caa536c4959d46171a078a6908fac02\": rpc error: code = NotFound desc = could not find container \"6720048b4ee28892faf8bbec09bddd8c5caa536c4959d46171a078a6908fac02\": container with ID starting with 6720048b4ee28892faf8bbec09bddd8c5caa536c4959d46171a078a6908fac02 not found: ID does not exist" Oct 29 23:04:17.927398 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.927377 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552"] Oct 29 23:04:17.931184 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.931164 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-tt552"] Oct 29 23:04:17.998306 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:17.998232 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2"] Oct 29 23:04:18.003215 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:18.003181 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-rnqr2"] Oct 29 23:04:18.042219 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:18.042168 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5"] Oct 29 23:04:18.045971 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:18.045941 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-0266dd-openshift-default-c89468fc-gcqz5"] Oct 29 23:04:18.202266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:18.201655 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39efdaf8-97f8-47fc-964c-4bc0daab505f" path="/var/lib/kubelet/pods/39efdaf8-97f8-47fc-964c-4bc0daab505f/volumes" Oct 29 23:04:18.202444 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:18.202372 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcc509cf-1d6b-4453-b46b-d756cd4ec293" path="/var/lib/kubelet/pods/dcc509cf-1d6b-4453-b46b-d756cd4ec293/volumes" Oct 29 23:04:18.203004 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:18.202973 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e778eb53-b230-451e-b7ae-3d8512366c59" path="/var/lib/kubelet/pods/e778eb53-b230-451e-b7ae-3d8512366c59/volumes" Oct 29 23:04:20.367275 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.367230 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd"] Oct 29 23:04:20.367825 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.367810 4821 cpu_manager.go:414] "containerMap: removing container" podUID="dcc509cf-1d6b-4453-b46b-d756cd4ec293" containerName="istio-proxy" Oct 29 23:04:20.367867 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.367827 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcc509cf-1d6b-4453-b46b-d756cd4ec293" containerName="istio-proxy" Oct 29 23:04:20.367867 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.367848 4821 cpu_manager.go:414] "containerMap: removing container" podUID="5bfcb45f-0e94-4e58-bc91-9157be0569ed" containerName="istio-proxy" Oct 29 23:04:20.367867 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.367854 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bfcb45f-0e94-4e58-bc91-9157be0569ed" containerName="istio-proxy" Oct 29 23:04:20.367979 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.367871 4821 cpu_manager.go:414] "containerMap: removing container" podUID="e778eb53-b230-451e-b7ae-3d8512366c59" containerName="istio-proxy" Oct 29 23:04:20.367979 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.367877 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e778eb53-b230-451e-b7ae-3d8512366c59" containerName="istio-proxy" Oct 29 23:04:20.367979 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.367899 4821 cpu_manager.go:414] "containerMap: removing container" podUID="39efdaf8-97f8-47fc-964c-4bc0daab505f" containerName="istio-proxy" Oct 29 23:04:20.367979 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.367904 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39efdaf8-97f8-47fc-964c-4bc0daab505f" containerName="istio-proxy" Oct 29 23:04:20.367979 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.367911 4821 cpu_manager.go:414] "containerMap: removing container" podUID="c582c536-c024-4bb5-a50e-8d78ca55f55f" containerName="revoke-keys" Oct 29 23:04:20.367979 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.367916 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c582c536-c024-4bb5-a50e-8d78ca55f55f" containerName="revoke-keys" Oct 29 23:04:20.368146 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.367997 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="e778eb53-b230-451e-b7ae-3d8512366c59" containerName="istio-proxy" Oct 29 23:04:20.368146 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.368011 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="39efdaf8-97f8-47fc-964c-4bc0daab505f" containerName="istio-proxy" Oct 29 23:04:20.368146 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.368016 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="c582c536-c024-4bb5-a50e-8d78ca55f55f" containerName="revoke-keys" Oct 29 23:04:20.368146 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.368025 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcc509cf-1d6b-4453-b46b-d756cd4ec293" containerName="istio-proxy" Oct 29 23:04:20.373575 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.373555 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" Oct 29 23:04:20.395137 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.394863 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-1b7950-serving-cert\"" Oct 29 23:04:20.401344 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.401309 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd"] Oct 29 23:04:20.453472 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.453439 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d648624c-86e0-4640-8e90-048cd2c02d19-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd\" (UID: \"d648624c-86e0-4640-8e90-048cd2c02d19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" Oct 29 23:04:20.453625 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.453590 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d648624c-86e0-4640-8e90-048cd2c02d19-metrics-certs\") pod \"maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd\" (UID: \"d648624c-86e0-4640-8e90-048cd2c02d19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" Oct 29 23:04:20.453685 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.453625 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d648624c-86e0-4640-8e90-048cd2c02d19-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd\" (UID: \"d648624c-86e0-4640-8e90-048cd2c02d19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" Oct 29 23:04:20.453845 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.453827 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqg2c\" (UniqueName: \"kubernetes.io/projected/d648624c-86e0-4640-8e90-048cd2c02d19-kube-api-access-cqg2c\") pod \"maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd\" (UID: \"d648624c-86e0-4640-8e90-048cd2c02d19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" Oct 29 23:04:20.561525 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.561485 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqg2c\" (UniqueName: \"kubernetes.io/projected/d648624c-86e0-4640-8e90-048cd2c02d19-kube-api-access-cqg2c\") pod \"maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd\" (UID: \"d648624c-86e0-4640-8e90-048cd2c02d19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" Oct 29 23:04:20.561710 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.561563 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d648624c-86e0-4640-8e90-048cd2c02d19-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd\" (UID: \"d648624c-86e0-4640-8e90-048cd2c02d19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" Oct 29 23:04:20.561710 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.561684 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d648624c-86e0-4640-8e90-048cd2c02d19-metrics-certs\") pod \"maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd\" (UID: \"d648624c-86e0-4640-8e90-048cd2c02d19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" Oct 29 23:04:20.561834 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.561719 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d648624c-86e0-4640-8e90-048cd2c02d19-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd\" (UID: \"d648624c-86e0-4640-8e90-048cd2c02d19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" Oct 29 23:04:20.561988 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:04:20.561966 4821 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-serving-cert: secret "maas-api-e2e-ipp-a-1b7950-serving-cert" not found Oct 29 23:04:20.562071 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:04:20.562049 4821 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d648624c-86e0-4640-8e90-048cd2c02d19-maas-api-tls podName:d648624c-86e0-4640-8e90-048cd2c02d19 nodeName:}" failed. No retries permitted until 2026-10-29 23:04:21.062026405 +0000 UTC m=+1315.002574296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d648624c-86e0-4640-8e90-048cd2c02d19-maas-api-tls") pod "maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" (UID: "d648624c-86e0-4640-8e90-048cd2c02d19") : secret "maas-api-e2e-ipp-a-1b7950-serving-cert" not found Oct 29 23:04:20.562496 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.562412 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d648624c-86e0-4640-8e90-048cd2c02d19-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd\" (UID: \"d648624c-86e0-4640-8e90-048cd2c02d19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" Oct 29 23:04:20.564936 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.564915 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d648624c-86e0-4640-8e90-048cd2c02d19-metrics-certs\") pod \"maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd\" (UID: \"d648624c-86e0-4640-8e90-048cd2c02d19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" Oct 29 23:04:20.592811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:20.592756 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqg2c\" (UniqueName: \"kubernetes.io/projected/d648624c-86e0-4640-8e90-048cd2c02d19-kube-api-access-cqg2c\") pod \"maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd\" (UID: \"d648624c-86e0-4640-8e90-048cd2c02d19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" Oct 29 23:04:21.070072 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:21.070033 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d648624c-86e0-4640-8e90-048cd2c02d19-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd\" (UID: \"d648624c-86e0-4640-8e90-048cd2c02d19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" Oct 29 23:04:21.072817 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:21.072788 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d648624c-86e0-4640-8e90-048cd2c02d19-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd\" (UID: \"d648624c-86e0-4640-8e90-048cd2c02d19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" Oct 29 23:04:21.290158 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:21.290111 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" Oct 29 23:04:21.675577 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:21.675544 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd"] Oct 29 23:04:21.676282 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:04:21.676249 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd648624c_86e0_4640_8e90_048cd2c02d19.slice/crio-62fcd5289d58be748bbf1a639a986798deae51ad067b57d66744e03af2a255f7 WatchSource:0}: Error finding container 62fcd5289d58be748bbf1a639a986798deae51ad067b57d66744e03af2a255f7: Status 404 returned error can't find the container with id 62fcd5289d58be748bbf1a639a986798deae51ad067b57d66744e03af2a255f7 Oct 29 23:04:21.678524 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:21.678501 4821 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:04:21.929185 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:21.929091 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" event={"ID":"d648624c-86e0-4640-8e90-048cd2c02d19","Type":"ContainerStarted","Data":"62fcd5289d58be748bbf1a639a986798deae51ad067b57d66744e03af2a255f7"} Oct 29 23:04:22.938938 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:22.938901 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" event={"ID":"d648624c-86e0-4640-8e90-048cd2c02d19","Type":"ContainerStarted","Data":"c24b43949b00e6245399d91accc86bd3299745c124aa5d06c51afe74a9df32f5"} Oct 29 23:04:22.939395 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:22.938976 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" Oct 29 23:04:23.163381 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:23.163335 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" podStartSLOduration=2.816001118 podStartE2EDuration="3.163319442s" totalImagesPullingTime="347.318324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:22.983405919 +0000 UTC m=+1316.923953839" watchObservedRunningTime="2026-10-29 23:04:23.163319442 +0000 UTC m=+1317.103867352" Oct 29 23:04:23.166237 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:23.166191 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp"] Oct 29 23:04:23.199637 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:23.199549 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz"] Oct 29 23:04:23.912755 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:04:23.912715 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:04:28.166569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:28.166499 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp" podUID="b740bdc5-90ed-42bd-a889-3582c45c8062" containerName="payload-processing" containerID="cri-o://f8d660957e40a9690ac49ba841b40b99d81753919dae8650ab9e6df9dd702481" gracePeriod=25 Oct 29 23:04:28.209123 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:28.203273 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz" podUID="8a5244fa-2331-48c6-b32f-3369cad8bd3c" containerName="payload-pre-processing" containerID="cri-o://a95e7b6e4bb759dd09bddb50f53906c220bdecbb1c2af1d96e8f25b7e1359346" gracePeriod=25 Oct 29 23:04:28.561706 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:28.561650 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp" Oct 29 23:04:28.722002 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:28.721976 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b740bdc5-90ed-42bd-a889-3582c45c8062-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b740bdc5-90ed-42bd-a889-3582c45c8062-plugins-config-volume\") pod \"b740bdc5-90ed-42bd-a889-3582c45c8062\" (UID: \"b740bdc5-90ed-42bd-a889-3582c45c8062\") " Oct 29 23:04:28.722235 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:28.722216 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b740bdc5-90ed-42bd-a889-3582c45c8062-kube-api-access-6h2lm\" (UniqueName: \"kubernetes.io/projected/b740bdc5-90ed-42bd-a889-3582c45c8062-kube-api-access-6h2lm\") pod \"b740bdc5-90ed-42bd-a889-3582c45c8062\" (UID: \"b740bdc5-90ed-42bd-a889-3582c45c8062\") " Oct 29 23:04:28.722414 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:28.722394 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b740bdc5-90ed-42bd-a889-3582c45c8062-plugins-config-volume" pod "b740bdc5-90ed-42bd-a889-3582c45c8062" (UID: "b740bdc5-90ed-42bd-a889-3582c45c8062"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:28.722873 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:28.722850 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b740bdc5-90ed-42bd-a889-3582c45c8062-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:28.724452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:28.724428 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b740bdc5-90ed-42bd-a889-3582c45c8062-kube-api-access-6h2lm" pod "b740bdc5-90ed-42bd-a889-3582c45c8062" (UID: "b740bdc5-90ed-42bd-a889-3582c45c8062"). InnerVolumeSpecName "kube-api-access-6h2lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:28.732563 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:28.732506 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz" Oct 29 23:04:28.823820 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:28.823784 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a5244fa-2331-48c6-b32f-3369cad8bd3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5244fa-2331-48c6-b32f-3369cad8bd3c-plugins-config-volume\") pod \"8a5244fa-2331-48c6-b32f-3369cad8bd3c\" (UID: \"8a5244fa-2331-48c6-b32f-3369cad8bd3c\") " Oct 29 23:04:28.824010 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:28.823848 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5244fa-2331-48c6-b32f-3369cad8bd3c-kube-api-access-64bqp\" (UniqueName: \"kubernetes.io/projected/8a5244fa-2331-48c6-b32f-3369cad8bd3c-kube-api-access-64bqp\") pod \"8a5244fa-2331-48c6-b32f-3369cad8bd3c\" (UID: \"8a5244fa-2331-48c6-b32f-3369cad8bd3c\") " Oct 29 23:04:28.824232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:28.824193 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a5244fa-2331-48c6-b32f-3369cad8bd3c-plugins-config-volume" pod "8a5244fa-2331-48c6-b32f-3369cad8bd3c" (UID: "8a5244fa-2331-48c6-b32f-3369cad8bd3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:28.824648 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:28.824617 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h2lm\" (UniqueName: \"kubernetes.io/projected/b740bdc5-90ed-42bd-a889-3582c45c8062-kube-api-access-6h2lm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:28.824753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:28.824649 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a5244fa-2331-48c6-b32f-3369cad8bd3c-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:28.826268 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:28.826244 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5244fa-2331-48c6-b32f-3369cad8bd3c-kube-api-access-64bqp" pod "8a5244fa-2331-48c6-b32f-3369cad8bd3c" (UID: "8a5244fa-2331-48c6-b32f-3369cad8bd3c"). InnerVolumeSpecName "kube-api-access-64bqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:28.925893 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:28.925852 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64bqp\" (UniqueName: \"kubernetes.io/projected/8a5244fa-2331-48c6-b32f-3369cad8bd3c-kube-api-access-64bqp\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:28.956757 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:28.956729 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" Oct 29 23:04:29.003443 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:29.003355 4821 generic.go:356] "Generic (PLEG): container finished" podID="8a5244fa-2331-48c6-b32f-3369cad8bd3c" containerID="a95e7b6e4bb759dd09bddb50f53906c220bdecbb1c2af1d96e8f25b7e1359346" exitCode=0 Oct 29 23:04:29.003443 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:29.003417 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz" Oct 29 23:04:29.003443 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:29.003434 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz" event={"ID":"8a5244fa-2331-48c6-b32f-3369cad8bd3c","Type":"ContainerDied","Data":"a95e7b6e4bb759dd09bddb50f53906c220bdecbb1c2af1d96e8f25b7e1359346"} Oct 29 23:04:29.003748 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:29.003498 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz" event={"ID":"8a5244fa-2331-48c6-b32f-3369cad8bd3c","Type":"ContainerDied","Data":"cf667644006e3e00f7277da6f03aa37d05e94676eb550d24e307cafe78f37b55"} Oct 29 23:04:29.003748 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:29.003530 4821 scope.go:122] "RemoveContainer" containerID="a95e7b6e4bb759dd09bddb50f53906c220bdecbb1c2af1d96e8f25b7e1359346" Oct 29 23:04:29.005153 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:29.005129 4821 generic.go:356] "Generic (PLEG): container finished" podID="b740bdc5-90ed-42bd-a889-3582c45c8062" containerID="f8d660957e40a9690ac49ba841b40b99d81753919dae8650ab9e6df9dd702481" exitCode=0 Oct 29 23:04:29.005360 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:29.005173 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp" Oct 29 23:04:29.005360 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:29.005185 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp" event={"ID":"b740bdc5-90ed-42bd-a889-3582c45c8062","Type":"ContainerDied","Data":"f8d660957e40a9690ac49ba841b40b99d81753919dae8650ab9e6df9dd702481"} Oct 29 23:04:29.005360 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:29.005243 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp" event={"ID":"b740bdc5-90ed-42bd-a889-3582c45c8062","Type":"ContainerDied","Data":"9b10d4a27d4360523d8a72f693c6894ec50cb6bd2e740f97ffe25b92697685e7"} Oct 29 23:04:29.018001 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:29.017979 4821 scope.go:122] "RemoveContainer" containerID="a95e7b6e4bb759dd09bddb50f53906c220bdecbb1c2af1d96e8f25b7e1359346" Oct 29 23:04:29.018365 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:04:29.018342 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a95e7b6e4bb759dd09bddb50f53906c220bdecbb1c2af1d96e8f25b7e1359346\": container with ID starting with a95e7b6e4bb759dd09bddb50f53906c220bdecbb1c2af1d96e8f25b7e1359346 not found: ID does not exist" containerID="a95e7b6e4bb759dd09bddb50f53906c220bdecbb1c2af1d96e8f25b7e1359346" Oct 29 23:04:29.018490 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:29.018376 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a95e7b6e4bb759dd09bddb50f53906c220bdecbb1c2af1d96e8f25b7e1359346"} err="failed to get container status \"a95e7b6e4bb759dd09bddb50f53906c220bdecbb1c2af1d96e8f25b7e1359346\": rpc error: code = NotFound desc = could not find container \"a95e7b6e4bb759dd09bddb50f53906c220bdecbb1c2af1d96e8f25b7e1359346\": container with ID starting with a95e7b6e4bb759dd09bddb50f53906c220bdecbb1c2af1d96e8f25b7e1359346 not found: ID does not exist" Oct 29 23:04:29.018490 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:29.018400 4821 scope.go:122] "RemoveContainer" containerID="f8d660957e40a9690ac49ba841b40b99d81753919dae8650ab9e6df9dd702481" Oct 29 23:04:29.030783 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:29.030755 4821 scope.go:122] "RemoveContainer" containerID="f8d660957e40a9690ac49ba841b40b99d81753919dae8650ab9e6df9dd702481" Oct 29 23:04:29.031153 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:04:29.031128 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8d660957e40a9690ac49ba841b40b99d81753919dae8650ab9e6df9dd702481\": container with ID starting with f8d660957e40a9690ac49ba841b40b99d81753919dae8650ab9e6df9dd702481 not found: ID does not exist" containerID="f8d660957e40a9690ac49ba841b40b99d81753919dae8650ab9e6df9dd702481" Oct 29 23:04:29.031438 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:29.031163 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8d660957e40a9690ac49ba841b40b99d81753919dae8650ab9e6df9dd702481"} err="failed to get container status \"f8d660957e40a9690ac49ba841b40b99d81753919dae8650ab9e6df9dd702481\": rpc error: code = NotFound desc = could not find container \"f8d660957e40a9690ac49ba841b40b99d81753919dae8650ab9e6df9dd702481\": container with ID starting with f8d660957e40a9690ac49ba841b40b99d81753919dae8650ab9e6df9dd702481 not found: ID does not exist" Oct 29 23:04:29.066638 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:29.065066 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp"] Oct 29 23:04:29.069133 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:29.069108 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1e1c46-87bbb5cfd-s6vzp"] Oct 29 23:04:29.105130 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:29.105102 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz"] Oct 29 23:04:29.108771 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:29.108747 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1e1c46-5cf9c55fd7-qfqnz"] Oct 29 23:04:30.170875 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:30.170823 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a5244fa-2331-48c6-b32f-3369cad8bd3c" path="/var/lib/kubelet/pods/8a5244fa-2331-48c6-b32f-3369cad8bd3c/volumes" Oct 29 23:04:30.171467 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:30.171446 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b740bdc5-90ed-42bd-a889-3582c45c8062" path="/var/lib/kubelet/pods/b740bdc5-90ed-42bd-a889-3582c45c8062/volumes" Oct 29 23:04:35.353427 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.353390 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9"] Oct 29 23:04:35.354188 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.354166 4821 cpu_manager.go:414] "containerMap: removing container" podUID="8a5244fa-2331-48c6-b32f-3369cad8bd3c" containerName="payload-pre-processing" Oct 29 23:04:35.354327 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.354188 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5244fa-2331-48c6-b32f-3369cad8bd3c" containerName="payload-pre-processing" Oct 29 23:04:35.354327 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.354247 4821 cpu_manager.go:414] "containerMap: removing container" podUID="b740bdc5-90ed-42bd-a889-3582c45c8062" containerName="payload-processing" Oct 29 23:04:35.354327 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.354256 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b740bdc5-90ed-42bd-a889-3582c45c8062" containerName="payload-processing" Oct 29 23:04:35.354504 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.354447 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5244fa-2331-48c6-b32f-3369cad8bd3c" containerName="payload-pre-processing" Oct 29 23:04:35.354504 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.354467 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="b740bdc5-90ed-42bd-a889-3582c45c8062" containerName="payload-processing" Oct 29 23:04:35.359815 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.359779 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:04:35.382632 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.382602 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9"] Oct 29 23:04:35.389068 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.389040 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-7d18f7\"/\"openshift-service-ca.crt\"" Oct 29 23:04:35.389285 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.389172 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-7d18f7\"/\"default-dockercfg-tbxjk\"" Oct 29 23:04:35.390770 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.390749 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-7d18f7\"/\"kube-root-ca.crt\"" Oct 29 23:04:35.390888 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.390813 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-7d18f7\"/\"ipp-route-7d18f7-kserve-self-signed-certs\"" Oct 29 23:04:35.428056 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.428013 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-home\") pod \"ipp-route-7d18f7-kserve-794d8647c-n5nv9\" (UID: \"d0417ea4-6fa4-4ab8-8087-634347f03002\") " pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:04:35.428234 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.428088 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-tmp-dir\") pod \"ipp-route-7d18f7-kserve-794d8647c-n5nv9\" (UID: \"d0417ea4-6fa4-4ab8-8087-634347f03002\") " pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:04:35.428234 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.428216 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hfmq\" (UniqueName: \"kubernetes.io/projected/d0417ea4-6fa4-4ab8-8087-634347f03002-kube-api-access-4hfmq\") pod \"ipp-route-7d18f7-kserve-794d8647c-n5nv9\" (UID: \"d0417ea4-6fa4-4ab8-8087-634347f03002\") " pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:04:35.428351 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.428287 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-dshm\") pod \"ipp-route-7d18f7-kserve-794d8647c-n5nv9\" (UID: \"d0417ea4-6fa4-4ab8-8087-634347f03002\") " pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:04:35.428351 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.428319 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0417ea4-6fa4-4ab8-8087-634347f03002-tls-certs\") pod \"ipp-route-7d18f7-kserve-794d8647c-n5nv9\" (UID: \"d0417ea4-6fa4-4ab8-8087-634347f03002\") " pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:04:35.428482 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.428375 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-model-cache\") pod \"ipp-route-7d18f7-kserve-794d8647c-n5nv9\" (UID: \"d0417ea4-6fa4-4ab8-8087-634347f03002\") " pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:04:35.530044 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.530005 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-home\") pod \"ipp-route-7d18f7-kserve-794d8647c-n5nv9\" (UID: \"d0417ea4-6fa4-4ab8-8087-634347f03002\") " pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:04:35.530276 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.530073 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-tmp-dir\") pod \"ipp-route-7d18f7-kserve-794d8647c-n5nv9\" (UID: \"d0417ea4-6fa4-4ab8-8087-634347f03002\") " pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:04:35.530276 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.530131 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hfmq\" (UniqueName: \"kubernetes.io/projected/d0417ea4-6fa4-4ab8-8087-634347f03002-kube-api-access-4hfmq\") pod \"ipp-route-7d18f7-kserve-794d8647c-n5nv9\" (UID: \"d0417ea4-6fa4-4ab8-8087-634347f03002\") " pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:04:35.530276 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.530185 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-dshm\") pod \"ipp-route-7d18f7-kserve-794d8647c-n5nv9\" (UID: \"d0417ea4-6fa4-4ab8-8087-634347f03002\") " pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:04:35.530469 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.530336 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0417ea4-6fa4-4ab8-8087-634347f03002-tls-certs\") pod \"ipp-route-7d18f7-kserve-794d8647c-n5nv9\" (UID: \"d0417ea4-6fa4-4ab8-8087-634347f03002\") " pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:04:35.530469 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.530422 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-model-cache\") pod \"ipp-route-7d18f7-kserve-794d8647c-n5nv9\" (UID: \"d0417ea4-6fa4-4ab8-8087-634347f03002\") " pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:04:35.530581 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.530478 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-tmp-dir\") pod \"ipp-route-7d18f7-kserve-794d8647c-n5nv9\" (UID: \"d0417ea4-6fa4-4ab8-8087-634347f03002\") " pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:04:35.530581 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.530522 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-home\") pod \"ipp-route-7d18f7-kserve-794d8647c-n5nv9\" (UID: \"d0417ea4-6fa4-4ab8-8087-634347f03002\") " pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:04:35.530759 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.530740 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-model-cache\") pod \"ipp-route-7d18f7-kserve-794d8647c-n5nv9\" (UID: \"d0417ea4-6fa4-4ab8-8087-634347f03002\") " pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:04:35.533075 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.533049 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0417ea4-6fa4-4ab8-8087-634347f03002-tls-certs\") pod \"ipp-route-7d18f7-kserve-794d8647c-n5nv9\" (UID: \"d0417ea4-6fa4-4ab8-8087-634347f03002\") " pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:04:35.533240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.533052 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-dshm\") pod \"ipp-route-7d18f7-kserve-794d8647c-n5nv9\" (UID: \"d0417ea4-6fa4-4ab8-8087-634347f03002\") " pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:04:35.554893 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.554866 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hfmq\" (UniqueName: \"kubernetes.io/projected/d0417ea4-6fa4-4ab8-8087-634347f03002-kube-api-access-4hfmq\") pod \"ipp-route-7d18f7-kserve-794d8647c-n5nv9\" (UID: \"d0417ea4-6fa4-4ab8-8087-634347f03002\") " pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:04:35.699269 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:35.684520 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:04:36.100781 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:36.100753 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9"] Oct 29 23:04:36.102094 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:04:36.102068 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0417ea4_6fa4_4ab8_8087_634347f03002.slice/crio-ac484c6dfd6e8275bcf0a76e7dca64a0b7eae441b3ec67d54419298a8699e212 WatchSource:0}: Error finding container ac484c6dfd6e8275bcf0a76e7dca64a0b7eae441b3ec67d54419298a8699e212: Status 404 returned error can't find the container with id ac484c6dfd6e8275bcf0a76e7dca64a0b7eae441b3ec67d54419298a8699e212 Oct 29 23:04:37.079654 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:37.079609 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" event={"ID":"d0417ea4-6fa4-4ab8-8087-634347f03002","Type":"ContainerStarted","Data":"cbb541155e8a90212889ed3a60c3c7ecb2e9c49f09977a8b15c25cbf24a8da00"} Oct 29 23:04:37.079654 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:37.079662 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" event={"ID":"d0417ea4-6fa4-4ab8-8087-634347f03002","Type":"ContainerStarted","Data":"ac484c6dfd6e8275bcf0a76e7dca64a0b7eae441b3ec67d54419298a8699e212"} Oct 29 23:04:37.125695 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:37.125636 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" podStartSLOduration=2.125620053 podStartE2EDuration="2.125620053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:37.125031025 +0000 UTC m=+1331.065578936" watchObservedRunningTime="2026-10-29 23:04:37.125620053 +0000 UTC m=+1331.066167963" Oct 29 23:04:37.685341 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:37.685315 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:04:43.695901 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:43.695864 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr"] Oct 29 23:04:43.705300 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:43.705268 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1e1c46-jq6mr"] Oct 29 23:04:44.228670 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:44.228636 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c582c536-c024-4bb5-a50e-8d78ca55f55f" path="/var/lib/kubelet/pods/c582c536-c024-4bb5-a50e-8d78ca55f55f/volumes" Oct 29 23:04:44.382821 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:44.382786 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg"] Oct 29 23:04:44.387806 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:44.387772 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-848fb8-l7bkg"] Oct 29 23:04:45.685535 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:45.685503 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:04:45.698799 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:45.698771 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:04:46.245267 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:46.240609 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aec37f6d-fd80-4f7c-99bd-6029d1d71471" path="/var/lib/kubelet/pods/aec37f6d-fd80-4f7c-99bd-6029d1d71471/volumes" Oct 29 23:04:46.294662 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:46.294629 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:04:48.968225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:48.953405 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv"] Oct 29 23:04:49.082739 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:49.082694 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv"] Oct 29 23:04:49.083100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:49.083072 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv" Oct 29 23:04:49.101220 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:49.101102 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-4f2e3a-dockercfg-gfrvh\"" Oct 29 23:04:49.113878 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:49.113850 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-4f2e3a\"" Oct 29 23:04:49.173982 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:49.173945 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4e8184-e3d8-40ba-865d-64f460d13942-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv\" (UID: \"7e4e8184-e3d8-40ba-865d-64f460d13942\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv" Oct 29 23:04:49.174217 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:49.174123 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbt2m\" (UniqueName: \"kubernetes.io/projected/7e4e8184-e3d8-40ba-865d-64f460d13942-kube-api-access-bbt2m\") pod \"payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv\" (UID: \"7e4e8184-e3d8-40ba-865d-64f460d13942\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv" Oct 29 23:04:49.284257 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:49.281291 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbt2m\" (UniqueName: \"kubernetes.io/projected/7e4e8184-e3d8-40ba-865d-64f460d13942-kube-api-access-bbt2m\") pod \"payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv\" (UID: \"7e4e8184-e3d8-40ba-865d-64f460d13942\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv" Oct 29 23:04:49.284257 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:49.281582 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4e8184-e3d8-40ba-865d-64f460d13942-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv\" (UID: \"7e4e8184-e3d8-40ba-865d-64f460d13942\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv" Oct 29 23:04:49.284257 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:49.283998 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4e8184-e3d8-40ba-865d-64f460d13942-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv\" (UID: \"7e4e8184-e3d8-40ba-865d-64f460d13942\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv" Oct 29 23:04:49.347228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:49.334007 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbt2m\" (UniqueName: \"kubernetes.io/projected/7e4e8184-e3d8-40ba-865d-64f460d13942-kube-api-access-bbt2m\") pod \"payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv\" (UID: \"7e4e8184-e3d8-40ba-865d-64f460d13942\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv" Oct 29 23:04:49.418569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:49.418527 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv" Oct 29 23:04:50.551414 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:50.551379 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv"] Oct 29 23:04:51.087516 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:51.087393 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq"] Oct 29 23:04:51.088327 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:51.088040 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" podUID="9e08c58b-702a-437f-bb87-1a4bc4727f78" containerName="istio-proxy" containerID="cri-o://f005b9644e366d8b8a2b67408b681da59e52d607ee2e3396ea815013bca3a01b" gracePeriod=30 Oct 29 23:04:51.450752 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:51.441261 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv" event={"ID":"7e4e8184-e3d8-40ba-865d-64f460d13942","Type":"ContainerStarted","Data":"1fca9d6c5075da2e273cc2ee5413fc5a39d2cda980e331b44987cdf1d6785045"} Oct 29 23:04:51.450752 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:51.441336 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv" event={"ID":"7e4e8184-e3d8-40ba-865d-64f460d13942","Type":"ContainerStarted","Data":"a54fed23c900166bc8331062905b6637f81535c129b6ec3f323f0d38ab66ff1e"} Oct 29 23:04:51.450752 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:51.442515 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv" Oct 29 23:04:51.523232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:51.507386 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv" podStartSLOduration=3.507361828 podStartE2EDuration="3.507361828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:51.494677243 +0000 UTC m=+1345.435225165" watchObservedRunningTime="2026-10-29 23:04:51.507361828 +0000 UTC m=+1345.447909739" Oct 29 23:04:54.075658 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:04:54.075613 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:04:54.733983 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:54.733943 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx"] Oct 29 23:04:54.740641 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:54.740608 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx" Oct 29 23:04:54.759357 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:54.758627 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:04:54.759357 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:54.759286 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:04:54.763496 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:54.763461 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx"] Oct 29 23:04:54.903315 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:54.902726 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjl8z\" (UniqueName: \"kubernetes.io/projected/150ec736-116e-4f11-b1be-b90c896896a4-kube-api-access-zjl8z\") pod \"maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx\" (UID: \"150ec736-116e-4f11-b1be-b90c896896a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx" Oct 29 23:04:54.903315 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:54.903005 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/150ec736-116e-4f11-b1be-b90c896896a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx\" (UID: \"150ec736-116e-4f11-b1be-b90c896896a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx" Oct 29 23:04:54.903315 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:54.903061 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/150ec736-116e-4f11-b1be-b90c896896a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx\" (UID: \"150ec736-116e-4f11-b1be-b90c896896a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx" Oct 29 23:04:55.007160 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:55.005997 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjl8z\" (UniqueName: \"kubernetes.io/projected/150ec736-116e-4f11-b1be-b90c896896a4-kube-api-access-zjl8z\") pod \"maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx\" (UID: \"150ec736-116e-4f11-b1be-b90c896896a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx" Oct 29 23:04:55.007160 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:55.006320 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/150ec736-116e-4f11-b1be-b90c896896a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx\" (UID: \"150ec736-116e-4f11-b1be-b90c896896a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx" Oct 29 23:04:55.007160 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:55.006376 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/150ec736-116e-4f11-b1be-b90c896896a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx\" (UID: \"150ec736-116e-4f11-b1be-b90c896896a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx" Oct 29 23:04:55.007160 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:55.007987 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/150ec736-116e-4f11-b1be-b90c896896a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx\" (UID: \"150ec736-116e-4f11-b1be-b90c896896a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx" Oct 29 23:04:55.020871 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:55.020842 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/150ec736-116e-4f11-b1be-b90c896896a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx\" (UID: \"150ec736-116e-4f11-b1be-b90c896896a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx" Oct 29 23:04:55.037223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:55.033722 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjl8z\" (UniqueName: \"kubernetes.io/projected/150ec736-116e-4f11-b1be-b90c896896a4-kube-api-access-zjl8z\") pod \"maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx\" (UID: \"150ec736-116e-4f11-b1be-b90c896896a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx" Oct 29 23:04:55.091024 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:55.085642 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx" Oct 29 23:04:55.457789 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:55.457759 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx"] Oct 29 23:04:55.458293 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:04:55.458266 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150ec736_116e_4f11_b1be_b90c896896a4.slice/crio-a1ebcba8a7ab1e4fe713e384cee5f64e347750e08527e1f533c7d3eadd97b277 WatchSource:0}: Error finding container a1ebcba8a7ab1e4fe713e384cee5f64e347750e08527e1f533c7d3eadd97b277: Status 404 returned error can't find the container with id a1ebcba8a7ab1e4fe713e384cee5f64e347750e08527e1f533c7d3eadd97b277 Oct 29 23:04:55.601835 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:55.601794 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx" event={"ID":"150ec736-116e-4f11-b1be-b90c896896a4","Type":"ContainerStarted","Data":"a1ebcba8a7ab1e4fe713e384cee5f64e347750e08527e1f533c7d3eadd97b277"} Oct 29 23:04:56.441182 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.441146 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:04:56.560763 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.560725 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-credential-socket\") pod \"9e08c58b-702a-437f-bb87-1a4bc4727f78\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " Oct 29 23:04:56.560950 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.560777 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-workload-socket\") pod \"9e08c58b-702a-437f-bb87-1a4bc4727f78\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " Oct 29 23:04:56.560950 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.560815 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-data\") pod \"9e08c58b-702a-437f-bb87-1a4bc4727f78\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " Oct 29 23:04:56.560950 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.560859 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e08c58b-702a-437f-bb87-1a4bc4727f78-kube-api-access-9ndwr\" (UniqueName: \"kubernetes.io/projected/9e08c58b-702a-437f-bb87-1a4bc4727f78-kube-api-access-9ndwr\") pod \"9e08c58b-702a-437f-bb87-1a4bc4727f78\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " Oct 29 23:04:56.560950 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.560919 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-envoy\") pod \"9e08c58b-702a-437f-bb87-1a4bc4727f78\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " Oct 29 23:04:56.561169 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.561025 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e08c58b-702a-437f-bb87-1a4bc4727f78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e08c58b-702a-437f-bb87-1a4bc4727f78-istiod-ca-cert\") pod \"9e08c58b-702a-437f-bb87-1a4bc4727f78\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " Oct 29 23:04:56.561169 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.561069 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-workload-certs\") pod \"9e08c58b-702a-437f-bb87-1a4bc4727f78\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " Oct 29 23:04:56.561169 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.561128 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-token\" (UniqueName: \"kubernetes.io/projected/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-token\") pod \"9e08c58b-702a-437f-bb87-1a4bc4727f78\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " Oct 29 23:04:56.561387 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.561264 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-podinfo\") pod \"9e08c58b-702a-437f-bb87-1a4bc4727f78\" (UID: \"9e08c58b-702a-437f-bb87-1a4bc4727f78\") " Oct 29 23:04:56.563706 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.563671 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e08c58b-702a-437f-bb87-1a4bc4727f78-istiod-ca-cert" pod "9e08c58b-702a-437f-bb87-1a4bc4727f78" (UID: "9e08c58b-702a-437f-bb87-1a4bc4727f78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:56.563973 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.563950 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-workload-certs" pod "9e08c58b-702a-437f-bb87-1a4bc4727f78" (UID: "9e08c58b-702a-437f-bb87-1a4bc4727f78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:56.564688 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.564646 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-credential-socket" pod "9e08c58b-702a-437f-bb87-1a4bc4727f78" (UID: "9e08c58b-702a-437f-bb87-1a4bc4727f78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:56.564966 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.564895 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-data" pod "9e08c58b-702a-437f-bb87-1a4bc4727f78" (UID: "9e08c58b-702a-437f-bb87-1a4bc4727f78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:56.565181 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.565148 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-workload-socket" pod "9e08c58b-702a-437f-bb87-1a4bc4727f78" (UID: "9e08c58b-702a-437f-bb87-1a4bc4727f78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:56.566763 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.566637 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-podinfo" pod "9e08c58b-702a-437f-bb87-1a4bc4727f78" (UID: "9e08c58b-702a-437f-bb87-1a4bc4727f78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:56.566996 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.566935 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-envoy" pod "9e08c58b-702a-437f-bb87-1a4bc4727f78" (UID: "9e08c58b-702a-437f-bb87-1a4bc4727f78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:56.567287 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.567008 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e08c58b-702a-437f-bb87-1a4bc4727f78-kube-api-access-9ndwr" pod "9e08c58b-702a-437f-bb87-1a4bc4727f78" (UID: "9e08c58b-702a-437f-bb87-1a4bc4727f78"). InnerVolumeSpecName "kube-api-access-9ndwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:56.568760 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.568716 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-token" pod "9e08c58b-702a-437f-bb87-1a4bc4727f78" (UID: "9e08c58b-702a-437f-bb87-1a4bc4727f78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:56.619249 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.616340 4821 generic.go:356] "Generic (PLEG): container finished" podID="9e08c58b-702a-437f-bb87-1a4bc4727f78" containerID="f005b9644e366d8b8a2b67408b681da59e52d607ee2e3396ea815013bca3a01b" exitCode=0 Oct 29 23:04:56.619249 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.616458 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" event={"ID":"9e08c58b-702a-437f-bb87-1a4bc4727f78","Type":"ContainerDied","Data":"f005b9644e366d8b8a2b67408b681da59e52d607ee2e3396ea815013bca3a01b"} Oct 29 23:04:56.619249 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.616514 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" event={"ID":"9e08c58b-702a-437f-bb87-1a4bc4727f78","Type":"ContainerDied","Data":"e79b2218fb5fc6fae2babbceed8cb39731f7a3d1e1c05fa85c4ef8076a9a95eb"} Oct 29 23:04:56.619249 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.616547 4821 scope.go:122] "RemoveContainer" containerID="f005b9644e366d8b8a2b67408b681da59e52d607ee2e3396ea815013bca3a01b" Oct 29 23:04:56.619249 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.616732 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq" Oct 29 23:04:56.623163 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.622998 4821 generic.go:356] "Generic (PLEG): container finished" podID="150ec736-116e-4f11-b1be-b90c896896a4" containerID="e411dda91e8ed10c78b320184303b79528172aaecb550597a37d5e391d609586" exitCode=0 Oct 29 23:04:56.623163 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.623086 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx" event={"ID":"150ec736-116e-4f11-b1be-b90c896896a4","Type":"ContainerDied","Data":"e411dda91e8ed10c78b320184303b79528172aaecb550597a37d5e391d609586"} Oct 29 23:04:56.641033 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.641003 4821 scope.go:122] "RemoveContainer" containerID="159bb96543d6f0d3c192fa136d44fbc179e9692be2614277723c82383d25b5fa" Oct 29 23:04:56.655303 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.655277 4821 scope.go:122] "RemoveContainer" containerID="f005b9644e366d8b8a2b67408b681da59e52d607ee2e3396ea815013bca3a01b" Oct 29 23:04:56.656056 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:04:56.656023 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f005b9644e366d8b8a2b67408b681da59e52d607ee2e3396ea815013bca3a01b\": container with ID starting with f005b9644e366d8b8a2b67408b681da59e52d607ee2e3396ea815013bca3a01b not found: ID does not exist" containerID="f005b9644e366d8b8a2b67408b681da59e52d607ee2e3396ea815013bca3a01b" Oct 29 23:04:56.656229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.656068 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f005b9644e366d8b8a2b67408b681da59e52d607ee2e3396ea815013bca3a01b"} err="failed to get container status \"f005b9644e366d8b8a2b67408b681da59e52d607ee2e3396ea815013bca3a01b\": rpc error: code = NotFound desc = could not find container \"f005b9644e366d8b8a2b67408b681da59e52d607ee2e3396ea815013bca3a01b\": container with ID starting with f005b9644e366d8b8a2b67408b681da59e52d607ee2e3396ea815013bca3a01b not found: ID does not exist" Oct 29 23:04:56.656229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.656090 4821 scope.go:122] "RemoveContainer" containerID="159bb96543d6f0d3c192fa136d44fbc179e9692be2614277723c82383d25b5fa" Oct 29 23:04:56.656486 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:04:56.656467 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"159bb96543d6f0d3c192fa136d44fbc179e9692be2614277723c82383d25b5fa\": container with ID starting with 159bb96543d6f0d3c192fa136d44fbc179e9692be2614277723c82383d25b5fa not found: ID does not exist" containerID="159bb96543d6f0d3c192fa136d44fbc179e9692be2614277723c82383d25b5fa" Oct 29 23:04:56.656553 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.656492 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"159bb96543d6f0d3c192fa136d44fbc179e9692be2614277723c82383d25b5fa"} err="failed to get container status \"159bb96543d6f0d3c192fa136d44fbc179e9692be2614277723c82383d25b5fa\": rpc error: code = NotFound desc = could not find container \"159bb96543d6f0d3c192fa136d44fbc179e9692be2614277723c82383d25b5fa\": container with ID starting with 159bb96543d6f0d3c192fa136d44fbc179e9692be2614277723c82383d25b5fa not found: ID does not exist" Oct 29 23:04:56.664434 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.664393 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:56.664434 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.664425 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:56.664434 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.664435 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:56.664669 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.664444 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ndwr\" (UniqueName: \"kubernetes.io/projected/9e08c58b-702a-437f-bb87-1a4bc4727f78-kube-api-access-9ndwr\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:56.664669 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.664453 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:56.664669 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.664469 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e08c58b-702a-437f-bb87-1a4bc4727f78-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:56.664669 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.664480 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e08c58b-702a-437f-bb87-1a4bc4727f78-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:56.664669 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.664488 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:56.664669 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.664495 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e08c58b-702a-437f-bb87-1a4bc4727f78-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:56.719987 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.719937 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq"] Oct 29 23:04:56.726293 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:56.726251 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-1e1c46-openshift-default-776f84546f-phsfq"] Oct 29 23:04:57.852598 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.852560 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl"] Oct 29 23:04:57.853529 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.853507 4821 cpu_manager.go:414] "containerMap: removing container" podUID="9e08c58b-702a-437f-bb87-1a4bc4727f78" containerName="istio-proxy" Oct 29 23:04:57.853585 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.853534 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e08c58b-702a-437f-bb87-1a4bc4727f78" containerName="istio-proxy" Oct 29 23:04:57.853585 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.853556 4821 cpu_manager.go:414] "containerMap: removing container" podUID="9e08c58b-702a-437f-bb87-1a4bc4727f78" containerName="istio-proxy" Oct 29 23:04:57.853585 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.853564 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e08c58b-702a-437f-bb87-1a4bc4727f78" containerName="istio-proxy" Oct 29 23:04:57.853780 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.853767 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e08c58b-702a-437f-bb87-1a4bc4727f78" containerName="istio-proxy" Oct 29 23:04:57.853815 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.853786 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e08c58b-702a-437f-bb87-1a4bc4727f78" containerName="istio-proxy" Oct 29 23:04:57.891078 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.891001 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx" Oct 29 23:04:57.905447 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.905416 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl"] Oct 29 23:04:57.905606 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.905556 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:57.987289 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.987252 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/150ec736-116e-4f11-b1be-b90c896896a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/150ec736-116e-4f11-b1be-b90c896896a4-maas-api-xks-ca\") pod \"150ec736-116e-4f11-b1be-b90c896896a4\" (UID: \"150ec736-116e-4f11-b1be-b90c896896a4\") " Oct 29 23:04:57.987521 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.987380 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/150ec736-116e-4f11-b1be-b90c896896a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/150ec736-116e-4f11-b1be-b90c896896a4-maas-api-openshift-ca\") pod \"150ec736-116e-4f11-b1be-b90c896896a4\" (UID: \"150ec736-116e-4f11-b1be-b90c896896a4\") " Oct 29 23:04:57.987521 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.987440 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/150ec736-116e-4f11-b1be-b90c896896a4-kube-api-access-zjl8z\" (UniqueName: \"kubernetes.io/projected/150ec736-116e-4f11-b1be-b90c896896a4-kube-api-access-zjl8z\") pod \"150ec736-116e-4f11-b1be-b90c896896a4\" (UID: \"150ec736-116e-4f11-b1be-b90c896896a4\") " Oct 29 23:04:57.988986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.987913 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-workload-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:57.988986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.987962 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914c88e0-ed28-42c6-b01d-1be633f49e4b-istiod-ca-cert\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:57.988986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.988065 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-workload-certs\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:57.988986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.988168 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-data\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:57.988986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.988292 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-podinfo\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:57.988986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.988455 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqdkg\" (UniqueName: \"kubernetes.io/projected/914c88e0-ed28-42c6-b01d-1be633f49e4b-kube-api-access-xqdkg\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:57.988986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.988500 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-credential-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:57.988986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.988633 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-envoy\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:57.988986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.988667 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-token\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:57.988986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.988713 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/150ec736-116e-4f11-b1be-b90c896896a4-maas-api-openshift-ca" pod "150ec736-116e-4f11-b1be-b90c896896a4" (UID: "150ec736-116e-4f11-b1be-b90c896896a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:57.988986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.988953 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/150ec736-116e-4f11-b1be-b90c896896a4-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:57.991707 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.991680 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/150ec736-116e-4f11-b1be-b90c896896a4-maas-api-xks-ca" pod "150ec736-116e-4f11-b1be-b90c896896a4" (UID: "150ec736-116e-4f11-b1be-b90c896896a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:57.992341 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:57.992320 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150ec736-116e-4f11-b1be-b90c896896a4-kube-api-access-zjl8z" pod "150ec736-116e-4f11-b1be-b90c896896a4" (UID: "150ec736-116e-4f11-b1be-b90c896896a4"). InnerVolumeSpecName "kube-api-access-zjl8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:58.090728 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.090689 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-envoy\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:58.090728 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.090734 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-token\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:58.091045 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.090813 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-workload-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:58.091045 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.090842 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914c88e0-ed28-42c6-b01d-1be633f49e4b-istiod-ca-cert\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:58.091045 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.090923 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-workload-certs\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:58.091045 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.091006 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-data\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:58.091345 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.091101 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-podinfo\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:58.091345 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.091155 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqdkg\" (UniqueName: \"kubernetes.io/projected/914c88e0-ed28-42c6-b01d-1be633f49e4b-kube-api-access-xqdkg\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:58.091345 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.091188 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-credential-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:58.091345 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.091268 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-workload-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:58.091345 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.091296 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/150ec736-116e-4f11-b1be-b90c896896a4-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:58.091345 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.091315 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjl8z\" (UniqueName: \"kubernetes.io/projected/150ec736-116e-4f11-b1be-b90c896896a4-kube-api-access-zjl8z\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:04:58.091676 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.091484 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-data\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:58.091676 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.091635 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914c88e0-ed28-42c6-b01d-1be633f49e4b-istiod-ca-cert\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:58.091789 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.091723 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-credential-socket\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:58.091789 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.091776 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-workload-certs\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:58.093680 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.093657 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-envoy\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:58.094045 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.094025 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-podinfo\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:58.115686 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.115603 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-token\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:58.116690 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.116668 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqdkg\" (UniqueName: \"kubernetes.io/projected/914c88e0-ed28-42c6-b01d-1be633f49e4b-kube-api-access-xqdkg\") pod \"e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:58.191611 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.191513 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e08c58b-702a-437f-bb87-1a4bc4727f78" path="/var/lib/kubelet/pods/9e08c58b-702a-437f-bb87-1a4bc4727f78/volumes" Oct 29 23:04:58.261342 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.261305 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:04:58.651813 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.650755 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx" event={"ID":"150ec736-116e-4f11-b1be-b90c896896a4","Type":"ContainerDied","Data":"a1ebcba8a7ab1e4fe713e384cee5f64e347750e08527e1f533c7d3eadd97b277"} Oct 29 23:04:58.651813 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.650823 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1ebcba8a7ab1e4fe713e384cee5f64e347750e08527e1f533c7d3eadd97b277" Oct 29 23:04:58.651813 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.650893 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx" Oct 29 23:04:58.733018 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.730580 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl"] Oct 29 23:04:58.744637 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:04:58.743591 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914c88e0_ed28_42c6_b01d_1be633f49e4b.slice/crio-7e9df837bb74aa6af0a8a99cc1a1acb32d753db02f870e946f78af4fc9887c73 WatchSource:0}: Error finding container 7e9df837bb74aa6af0a8a99cc1a1acb32d753db02f870e946f78af4fc9887c73: Status 404 returned error can't find the container with id 7e9df837bb74aa6af0a8a99cc1a1acb32d753db02f870e946f78af4fc9887c73 Oct 29 23:04:58.749104 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.748057 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:04:58.749104 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.748154 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:04:58.749104 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.748225 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:04:58.827238 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.825322 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:04:58.827238 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:58.825470 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" podUID="d654f3f3-8c6b-46e4-b528-0459173f9d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:04:59.169065 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:59.169034 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e8612f6-84jxm"] Oct 29 23:04:59.169680 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:59.169660 4821 cpu_manager.go:414] "containerMap: removing container" podUID="150ec736-116e-4f11-b1be-b90c896896a4" containerName="revoke-keys" Oct 29 23:04:59.169742 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:59.169683 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="150ec736-116e-4f11-b1be-b90c896896a4" containerName="revoke-keys" Oct 29 23:04:59.169901 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:59.169887 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="150ec736-116e-4f11-b1be-b90c896896a4" containerName="revoke-keys" Oct 29 23:04:59.188594 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:59.188411 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8612f6-84jxm" Oct 29 23:04:59.198456 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:59.198426 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e8612f6-84jxm"] Oct 29 23:04:59.208859 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:59.208665 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:04:59.208859 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:59.208830 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:04:59.356064 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:59.356026 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448249df-154f-42be-9621-28d77dc52679-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e8612f6-84jxm\" (UID: \"448249df-154f-42be-9621-28d77dc52679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8612f6-84jxm" Oct 29 23:04:59.356322 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:59.356144 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bdvs\" (UniqueName: \"kubernetes.io/projected/448249df-154f-42be-9621-28d77dc52679-kube-api-access-7bdvs\") pod \"maas-api-revoke-sub-8e8612f6-84jxm\" (UID: \"448249df-154f-42be-9621-28d77dc52679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8612f6-84jxm" Oct 29 23:04:59.356322 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:59.356251 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448249df-154f-42be-9621-28d77dc52679-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e8612f6-84jxm\" (UID: \"448249df-154f-42be-9621-28d77dc52679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8612f6-84jxm" Oct 29 23:04:59.458117 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:59.458012 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448249df-154f-42be-9621-28d77dc52679-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e8612f6-84jxm\" (UID: \"448249df-154f-42be-9621-28d77dc52679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8612f6-84jxm" Oct 29 23:04:59.458349 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:59.458118 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bdvs\" (UniqueName: \"kubernetes.io/projected/448249df-154f-42be-9621-28d77dc52679-kube-api-access-7bdvs\") pod \"maas-api-revoke-sub-8e8612f6-84jxm\" (UID: \"448249df-154f-42be-9621-28d77dc52679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8612f6-84jxm" Oct 29 23:04:59.458349 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:59.458268 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448249df-154f-42be-9621-28d77dc52679-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e8612f6-84jxm\" (UID: \"448249df-154f-42be-9621-28d77dc52679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8612f6-84jxm" Oct 29 23:04:59.458705 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:59.458683 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448249df-154f-42be-9621-28d77dc52679-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e8612f6-84jxm\" (UID: \"448249df-154f-42be-9621-28d77dc52679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8612f6-84jxm" Oct 29 23:04:59.461156 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:59.461131 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448249df-154f-42be-9621-28d77dc52679-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e8612f6-84jxm\" (UID: \"448249df-154f-42be-9621-28d77dc52679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8612f6-84jxm" Oct 29 23:04:59.483891 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:59.483861 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bdvs\" (UniqueName: \"kubernetes.io/projected/448249df-154f-42be-9621-28d77dc52679-kube-api-access-7bdvs\") pod \"maas-api-revoke-sub-8e8612f6-84jxm\" (UID: \"448249df-154f-42be-9621-28d77dc52679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8612f6-84jxm" Oct 29 23:04:59.529113 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:59.529078 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8612f6-84jxm" Oct 29 23:04:59.666365 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:59.666246 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" event={"ID":"914c88e0-ed28-42c6-b01d-1be633f49e4b","Type":"ContainerStarted","Data":"c7790336fb3c251d1bc383f9d9109d8673895974bceb5234bc408391ce74a67c"} Oct 29 23:04:59.666365 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:59.666309 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" event={"ID":"914c88e0-ed28-42c6-b01d-1be633f49e4b","Type":"ContainerStarted","Data":"7e9df837bb74aa6af0a8a99cc1a1acb32d753db02f870e946f78af4fc9887c73"} Oct 29 23:04:59.719285 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:59.718780 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" podStartSLOduration=2.7187593 podStartE2EDuration="2.7187593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:59.714327853 +0000 UTC m=+1353.654875766" watchObservedRunningTime="2026-10-29 23:04:59.7187593 +0000 UTC m=+1353.659307211" Oct 29 23:04:59.864932 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:04:59.864902 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e8612f6-84jxm"] Oct 29 23:04:59.866158 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:04:59.866127 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod448249df_154f_42be_9621_28d77dc52679.slice/crio-6cde906b2da4f217aa92ca178e5eeefc36b47737dde5fa3eb20f2be0d054f44c WatchSource:0}: Error finding container 6cde906b2da4f217aa92ca178e5eeefc36b47737dde5fa3eb20f2be0d054f44c: Status 404 returned error can't find the container with id 6cde906b2da4f217aa92ca178e5eeefc36b47737dde5fa3eb20f2be0d054f44c Oct 29 23:05:00.039286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.037214 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp"] Oct 29 23:05:00.043471 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.042634 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp" Oct 29 23:05:00.059834 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.059571 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-3f7d4b\"" Oct 29 23:05:00.059834 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.059743 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-3f7d4b-dockercfg-mt55q\"" Oct 29 23:05:00.064555 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.064529 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp"] Oct 29 23:05:00.195656 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.181438 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c03888c-f9e6-4f33-95bb-e17e22d4ff54-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp\" (UID: \"7c03888c-f9e6-4f33-95bb-e17e22d4ff54\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp" Oct 29 23:05:00.220635 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.196581 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7v7d\" (UniqueName: \"kubernetes.io/projected/7c03888c-f9e6-4f33-95bb-e17e22d4ff54-kube-api-access-k7v7d\") pod \"payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp\" (UID: \"7c03888c-f9e6-4f33-95bb-e17e22d4ff54\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp" Oct 29 23:05:00.282998 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.282960 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:05:00.284449 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.284420 4821 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 29 23:05:00.284619 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.284477 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" podUID="914c88e0-ed28-42c6-b01d-1be633f49e4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 29 23:05:00.338877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.336999 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7v7d\" (UniqueName: \"kubernetes.io/projected/7c03888c-f9e6-4f33-95bb-e17e22d4ff54-kube-api-access-k7v7d\") pod \"payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp\" (UID: \"7c03888c-f9e6-4f33-95bb-e17e22d4ff54\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp" Oct 29 23:05:00.338877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.337224 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c03888c-f9e6-4f33-95bb-e17e22d4ff54-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp\" (UID: \"7c03888c-f9e6-4f33-95bb-e17e22d4ff54\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp" Oct 29 23:05:00.338877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.338370 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c03888c-f9e6-4f33-95bb-e17e22d4ff54-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp\" (UID: \"7c03888c-f9e6-4f33-95bb-e17e22d4ff54\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp" Oct 29 23:05:00.363712 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.363649 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7v7d\" (UniqueName: \"kubernetes.io/projected/7c03888c-f9e6-4f33-95bb-e17e22d4ff54-kube-api-access-k7v7d\") pod \"payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp\" (UID: \"7c03888c-f9e6-4f33-95bb-e17e22d4ff54\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp" Oct 29 23:05:00.375906 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.375641 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp" Oct 29 23:05:00.513903 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.513872 4821 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 29 23:05:00.514091 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.513922 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" podUID="f1840a05-a8fe-40e2-b7f9-39a91f6f355f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 29 23:05:00.678975 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.678940 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8612f6-84jxm" event={"ID":"448249df-154f-42be-9621-28d77dc52679","Type":"ContainerStarted","Data":"fb72d757cabcc0f1fa3c196a1ab2670a80028107420cf53e3647a6d230970534"} Oct 29 23:05:00.679162 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.678994 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8612f6-84jxm" event={"ID":"448249df-154f-42be-9621-28d77dc52679","Type":"ContainerStarted","Data":"6cde906b2da4f217aa92ca178e5eeefc36b47737dde5fa3eb20f2be0d054f44c"} Oct 29 23:05:00.681258 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.681233 4821 generic.go:356] "Generic (PLEG): container finished" podID="d654f3f3-8c6b-46e4-b528-0459173f9d2a" containerID="77c8d1d1c29d955bf8d7e8e1b2df5a3fad1b59108a158739d1ccadd269372b55" exitCode=0 Oct 29 23:05:00.681384 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.681258 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" event={"ID":"d654f3f3-8c6b-46e4-b528-0459173f9d2a","Type":"ContainerDied","Data":"77c8d1d1c29d955bf8d7e8e1b2df5a3fad1b59108a158739d1ccadd269372b55"} Oct 29 23:05:00.681726 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.681707 4821 scope.go:122] "RemoveContainer" containerID="77c8d1d1c29d955bf8d7e8e1b2df5a3fad1b59108a158739d1ccadd269372b55" Oct 29 23:05:00.683596 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.683570 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:05:00.683813 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.683651 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:05:00.683813 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.683703 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:05:00.684919 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.684893 4821 generic.go:356] "Generic (PLEG): container finished" podID="f1840a05-a8fe-40e2-b7f9-39a91f6f355f" containerID="dc0623074d766ff1f978853f514e08c7f9386d8f86c3c52ce1f735aba831b7be" exitCode=0 Oct 29 23:05:00.685010 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.684962 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" event={"ID":"f1840a05-a8fe-40e2-b7f9-39a91f6f355f","Type":"ContainerDied","Data":"dc0623074d766ff1f978853f514e08c7f9386d8f86c3c52ce1f735aba831b7be"} Oct 29 23:05:00.685311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.685293 4821 scope.go:122] "RemoveContainer" containerID="dc0623074d766ff1f978853f514e08c7f9386d8f86c3c52ce1f735aba831b7be" Oct 29 23:05:00.686977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.686948 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:05:00.687100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.687029 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:05:00.687146 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.687096 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:05:00.746382 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.746318 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8612f6-84jxm" podStartSLOduration=1.7462964539999999 podStartE2EDuration="1.746296454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:00.74544308 +0000 UTC m=+1354.685991000" watchObservedRunningTime="2026-10-29 23:05:00.746296454 +0000 UTC m=+1354.686844366" Oct 29 23:05:00.802851 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:00.802821 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp"] Oct 29 23:05:00.804741 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:05:00.804717 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c03888c_f9e6_4f33_95bb_e17e22d4ff54.slice/crio-df789ee262939d3f33b490cbf144e7381c30198e03e6d8925c3e66861266db62 WatchSource:0}: Error finding container df789ee262939d3f33b490cbf144e7381c30198e03e6d8925c3e66861266db62: Status 404 returned error can't find the container with id df789ee262939d3f33b490cbf144e7381c30198e03e6d8925c3e66861266db62 Oct 29 23:05:01.265239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:01.265190 4821 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 29 23:05:01.265644 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:01.265251 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" podUID="914c88e0-ed28-42c6-b01d-1be633f49e4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 29 23:05:01.697836 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:01.697078 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" event={"ID":"d654f3f3-8c6b-46e4-b528-0459173f9d2a","Type":"ContainerStarted","Data":"02227b5210bf0c78963aa280342b6533c0b18c672d0cba6c3d8935f546982b3f"} Oct 29 23:05:01.699438 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:01.699404 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" event={"ID":"f1840a05-a8fe-40e2-b7f9-39a91f6f355f","Type":"ContainerStarted","Data":"7e9cb7a0b247a2a1ce0018539a2a1ef4ba002451743af20747de0a6fa7c5c5d6"} Oct 29 23:05:01.701140 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:01.701117 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp" event={"ID":"7c03888c-f9e6-4f33-95bb-e17e22d4ff54","Type":"ContainerStarted","Data":"0fdcbaa06d609e99160aeed362815f54fddec044d1539a5ab897b200bc301f74"} Oct 29 23:05:01.701262 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:01.701159 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp" event={"ID":"7c03888c-f9e6-4f33-95bb-e17e22d4ff54","Type":"ContainerStarted","Data":"df789ee262939d3f33b490cbf144e7381c30198e03e6d8925c3e66861266db62"} Oct 29 23:05:01.701262 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:01.701189 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp" Oct 29 23:05:01.702637 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:01.702613 4821 generic.go:356] "Generic (PLEG): container finished" podID="448249df-154f-42be-9621-28d77dc52679" containerID="fb72d757cabcc0f1fa3c196a1ab2670a80028107420cf53e3647a6d230970534" exitCode=0 Oct 29 23:05:01.702738 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:01.702663 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8612f6-84jxm" event={"ID":"448249df-154f-42be-9621-28d77dc52679","Type":"ContainerDied","Data":"fb72d757cabcc0f1fa3c196a1ab2670a80028107420cf53e3647a6d230970534"} Oct 29 23:05:01.833936 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:01.833829 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:05:01.836397 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:01.835757 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:05:01.836397 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:01.835810 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" podUID="d654f3f3-8c6b-46e4-b528-0459173f9d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 29 23:05:01.862982 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:01.862947 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:05:01.863378 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:01.863356 4821 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 29 23:05:01.863568 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:01.863543 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" podUID="f1840a05-a8fe-40e2-b7f9-39a91f6f355f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 29 23:05:01.962280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:01.961370 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp" podStartSLOduration=1.961345066 podStartE2EDuration="1.961345066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:01.935779984 +0000 UTC m=+1355.876327890" watchObservedRunningTime="2026-10-29 23:05:01.961345066 +0000 UTC m=+1355.901892981" Oct 29 23:05:02.265671 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:02.265575 4821 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 29 23:05:02.266299 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:02.266266 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" podUID="914c88e0-ed28-42c6-b01d-1be633f49e4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 29 23:05:02.838417 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:02.838380 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:05:02.838684 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:02.838658 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" podUID="d654f3f3-8c6b-46e4-b528-0459173f9d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 29 23:05:02.863603 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:02.863568 4821 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 29 23:05:02.863863 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:02.863839 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" podUID="f1840a05-a8fe-40e2-b7f9-39a91f6f355f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 29 23:05:03.309969 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:03.309931 4821 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 29 23:05:03.310550 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:03.310520 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" podUID="914c88e0-ed28-42c6-b01d-1be633f49e4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 29 23:05:03.679226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:03.676616 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv" Oct 29 23:05:03.717410 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:03.717378 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8612f6-84jxm" Oct 29 23:05:03.885298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:03.844523 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:05:03.885298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:03.844587 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" podUID="d654f3f3-8c6b-46e4-b528-0459173f9d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 29 23:05:03.885298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:03.845451 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/448249df-154f-42be-9621-28d77dc52679-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448249df-154f-42be-9621-28d77dc52679-maas-api-xks-ca\") pod \"448249df-154f-42be-9621-28d77dc52679\" (UID: \"448249df-154f-42be-9621-28d77dc52679\") " Oct 29 23:05:03.885298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:03.845510 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/448249df-154f-42be-9621-28d77dc52679-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448249df-154f-42be-9621-28d77dc52679-maas-api-openshift-ca\") pod \"448249df-154f-42be-9621-28d77dc52679\" (UID: \"448249df-154f-42be-9621-28d77dc52679\") " Oct 29 23:05:03.885298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:03.845582 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/448249df-154f-42be-9621-28d77dc52679-kube-api-access-7bdvs\" (UniqueName: \"kubernetes.io/projected/448249df-154f-42be-9621-28d77dc52679-kube-api-access-7bdvs\") pod \"448249df-154f-42be-9621-28d77dc52679\" (UID: \"448249df-154f-42be-9621-28d77dc52679\") " Oct 29 23:05:03.885298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:03.848228 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/448249df-154f-42be-9621-28d77dc52679-maas-api-openshift-ca" pod "448249df-154f-42be-9621-28d77dc52679" (UID: "448249df-154f-42be-9621-28d77dc52679"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:03.975227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:03.971393 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448249df-154f-42be-9621-28d77dc52679-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:04.065309 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:04.062670 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/448249df-154f-42be-9621-28d77dc52679-kube-api-access-7bdvs" pod "448249df-154f-42be-9621-28d77dc52679" (UID: "448249df-154f-42be-9621-28d77dc52679"). InnerVolumeSpecName "kube-api-access-7bdvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:04.108055 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:04.108017 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/448249df-154f-42be-9621-28d77dc52679-maas-api-xks-ca" pod "448249df-154f-42be-9621-28d77dc52679" (UID: "448249df-154f-42be-9621-28d77dc52679"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:04.108693 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:04.108662 4821 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 29 23:05:04.108948 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:04.108922 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" podUID="f1840a05-a8fe-40e2-b7f9-39a91f6f355f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 29 23:05:04.109297 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:04.108716 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448249df-154f-42be-9621-28d77dc52679-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:04.109392 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:04.109316 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bdvs\" (UniqueName: \"kubernetes.io/projected/448249df-154f-42be-9621-28d77dc52679-kube-api-access-7bdvs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:04.111778 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:04.111749 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8612f6-84jxm" event={"ID":"448249df-154f-42be-9621-28d77dc52679","Type":"ContainerDied","Data":"6cde906b2da4f217aa92ca178e5eeefc36b47737dde5fa3eb20f2be0d054f44c"} Oct 29 23:05:04.111995 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:04.111980 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cde906b2da4f217aa92ca178e5eeefc36b47737dde5fa3eb20f2be0d054f44c" Oct 29 23:05:04.112183 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:04.112172 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8612f6-84jxm" Oct 29 23:05:04.300295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:04.267512 4821 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 29 23:05:04.300295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:04.267588 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" podUID="914c88e0-ed28-42c6-b01d-1be633f49e4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 29 23:05:04.839227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:04.838322 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:05:04.839227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:04.838390 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" podUID="d654f3f3-8c6b-46e4-b528-0459173f9d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 29 23:05:04.894100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:04.893990 4821 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 29 23:05:04.894100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:04.894044 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" podUID="f1840a05-a8fe-40e2-b7f9-39a91f6f355f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 29 23:05:05.284232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:05.274759 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn"] Oct 29 23:05:05.301692 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:05.287747 4821 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 29 23:05:05.301967 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:05.301937 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" podUID="914c88e0-ed28-42c6-b01d-1be633f49e4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 29 23:05:05.310499 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:05.310467 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g"] Oct 29 23:05:05.832224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:05.831718 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:05:05.832224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:05.831797 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" podUID="d654f3f3-8c6b-46e4-b528-0459173f9d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 29 23:05:05.866225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:05.865396 4821 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 29 23:05:05.866225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:05.865476 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" podUID="f1840a05-a8fe-40e2-b7f9-39a91f6f355f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 29 23:05:06.422630 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:06.422590 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:05:06.424188 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:06.424168 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:05:06.432712 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:06.428422 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:05:06.832286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:06.831648 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:05:06.832286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:06.831725 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" podUID="d654f3f3-8c6b-46e4-b528-0459173f9d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 29 23:05:06.864612 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:06.864576 4821 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 29 23:05:06.864877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:06.864848 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" podUID="f1840a05-a8fe-40e2-b7f9-39a91f6f355f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 29 23:05:07.837227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:07.832319 4821 patch_prober.go:29] interesting pod/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 29 23:05:07.837227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:07.832416 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" podUID="d654f3f3-8c6b-46e4-b528-0459173f9d2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 29 23:05:07.862283 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:07.862252 4821 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 29 23:05:07.862446 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:07.862303 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" podUID="f1840a05-a8fe-40e2-b7f9-39a91f6f355f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 29 23:05:08.837731 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:08.837701 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:05:08.838135 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:08.838090 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:05:08.838938 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:08.838919 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:05:08.867245 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:08.867223 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:05:08.867610 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:08.867587 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:05:08.868476 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:08.868458 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:05:09.412423 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.412381 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4"] Oct 29 23:05:09.413594 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.413572 4821 cpu_manager.go:414] "containerMap: removing container" podUID="448249df-154f-42be-9621-28d77dc52679" containerName="revoke-keys" Oct 29 23:05:09.413743 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.413731 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="448249df-154f-42be-9621-28d77dc52679" containerName="revoke-keys" Oct 29 23:05:09.413982 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.413972 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="448249df-154f-42be-9621-28d77dc52679" containerName="revoke-keys" Oct 29 23:05:09.418354 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.418331 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:05:09.422088 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.422039 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4"] Oct 29 23:05:09.437631 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.437594 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-d67deb\"/\"praxis-route-d67deb-kserve-self-signed-certs\"" Oct 29 23:05:09.437796 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.437725 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-d67deb\"/\"openshift-service-ca.crt\"" Oct 29 23:05:09.437873 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.437833 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-d67deb\"/\"default-dockercfg-bh5qp\"" Oct 29 23:05:09.438315 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.437946 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-d67deb\"/\"kube-root-ca.crt\"" Oct 29 23:05:09.530150 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.530116 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-dshm\") pod \"praxis-route-d67deb-kserve-5984778997-dx2v4\" (UID: \"0a560010-b749-4e0d-875c-4b5d880410ef\") " pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:05:09.530352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.530179 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-model-cache\") pod \"praxis-route-d67deb-kserve-5984778997-dx2v4\" (UID: \"0a560010-b749-4e0d-875c-4b5d880410ef\") " pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:05:09.530415 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.530383 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hns86\" (UniqueName: \"kubernetes.io/projected/0a560010-b749-4e0d-875c-4b5d880410ef-kube-api-access-hns86\") pod \"praxis-route-d67deb-kserve-5984778997-dx2v4\" (UID: \"0a560010-b749-4e0d-875c-4b5d880410ef\") " pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:05:09.530623 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.530596 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-tmp-dir\") pod \"praxis-route-d67deb-kserve-5984778997-dx2v4\" (UID: \"0a560010-b749-4e0d-875c-4b5d880410ef\") " pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:05:09.530682 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.530650 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-home\") pod \"praxis-route-d67deb-kserve-5984778997-dx2v4\" (UID: \"0a560010-b749-4e0d-875c-4b5d880410ef\") " pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:05:09.530765 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.530747 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a560010-b749-4e0d-875c-4b5d880410ef-tls-certs\") pod \"praxis-route-d67deb-kserve-5984778997-dx2v4\" (UID: \"0a560010-b749-4e0d-875c-4b5d880410ef\") " pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:05:09.632422 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.632385 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-dshm\") pod \"praxis-route-d67deb-kserve-5984778997-dx2v4\" (UID: \"0a560010-b749-4e0d-875c-4b5d880410ef\") " pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:05:09.632626 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.632448 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-model-cache\") pod \"praxis-route-d67deb-kserve-5984778997-dx2v4\" (UID: \"0a560010-b749-4e0d-875c-4b5d880410ef\") " pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:05:09.632626 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.632510 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hns86\" (UniqueName: \"kubernetes.io/projected/0a560010-b749-4e0d-875c-4b5d880410ef-kube-api-access-hns86\") pod \"praxis-route-d67deb-kserve-5984778997-dx2v4\" (UID: \"0a560010-b749-4e0d-875c-4b5d880410ef\") " pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:05:09.632756 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.632666 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-tmp-dir\") pod \"praxis-route-d67deb-kserve-5984778997-dx2v4\" (UID: \"0a560010-b749-4e0d-875c-4b5d880410ef\") " pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:05:09.632811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.632770 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-home\") pod \"praxis-route-d67deb-kserve-5984778997-dx2v4\" (UID: \"0a560010-b749-4e0d-875c-4b5d880410ef\") " pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:05:09.632935 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.632884 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a560010-b749-4e0d-875c-4b5d880410ef-tls-certs\") pod \"praxis-route-d67deb-kserve-5984778997-dx2v4\" (UID: \"0a560010-b749-4e0d-875c-4b5d880410ef\") " pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:05:09.633002 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.632943 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-model-cache\") pod \"praxis-route-d67deb-kserve-5984778997-dx2v4\" (UID: \"0a560010-b749-4e0d-875c-4b5d880410ef\") " pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:05:09.633389 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.633366 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-home\") pod \"praxis-route-d67deb-kserve-5984778997-dx2v4\" (UID: \"0a560010-b749-4e0d-875c-4b5d880410ef\") " pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:05:09.633554 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.633463 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-tmp-dir\") pod \"praxis-route-d67deb-kserve-5984778997-dx2v4\" (UID: \"0a560010-b749-4e0d-875c-4b5d880410ef\") " pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:05:09.635415 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.635392 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-dshm\") pod \"praxis-route-d67deb-kserve-5984778997-dx2v4\" (UID: \"0a560010-b749-4e0d-875c-4b5d880410ef\") " pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:05:09.635970 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.635953 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a560010-b749-4e0d-875c-4b5d880410ef-tls-certs\") pod \"praxis-route-d67deb-kserve-5984778997-dx2v4\" (UID: \"0a560010-b749-4e0d-875c-4b5d880410ef\") " pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:05:09.666710 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.666595 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hns86\" (UniqueName: \"kubernetes.io/projected/0a560010-b749-4e0d-875c-4b5d880410ef-kube-api-access-hns86\") pod \"praxis-route-d67deb-kserve-5984778997-dx2v4\" (UID: \"0a560010-b749-4e0d-875c-4b5d880410ef\") " pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:05:09.758990 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:09.757661 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:05:10.186808 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:10.186780 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4"] Oct 29 23:05:10.187436 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:05:10.187414 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a560010_b749_4e0d_875c_4b5d880410ef.slice/crio-428822774d3a01f5bcf2830f005574930854e6df14115034276c8a9d020bcfe4 WatchSource:0}: Error finding container 428822774d3a01f5bcf2830f005574930854e6df14115034276c8a9d020bcfe4: Status 404 returned error can't find the container with id 428822774d3a01f5bcf2830f005574930854e6df14115034276c8a9d020bcfe4 Oct 29 23:05:10.301954 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:10.301892 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn" podUID="1425a1f4-640b-4c45-bae9-7e0d4b66afcf" containerName="payload-processing" containerID="cri-o://a5133bbaeea0e3e64a6bf89d5f86db87684fc323dd3d4ca9c0c0544a27aa007a" gracePeriod=25 Oct 29 23:05:10.304291 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:10.304256 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" event={"ID":"0a560010-b749-4e0d-875c-4b5d880410ef","Type":"ContainerStarted","Data":"88e5d6e180ce3947b2d2a842a86e51aa2def76b0962b9e1acfd78620b9270b15"} Oct 29 23:05:10.304410 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:10.304305 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" event={"ID":"0a560010-b749-4e0d-875c-4b5d880410ef","Type":"ContainerStarted","Data":"428822774d3a01f5bcf2830f005574930854e6df14115034276c8a9d020bcfe4"} Oct 29 23:05:10.311985 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:10.311950 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g" podUID="a84143c5-852c-4ebb-8aee-c5e3dd323ef6" containerName="payload-pre-processing" containerID="cri-o://dde36c2a524b2e1da1ec2a6b280d5d00dec65e1bcc0adde2d9c159677a8df955" gracePeriod=25 Oct 29 23:05:10.352086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:10.352036 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" podStartSLOduration=1.3520226100000001 podStartE2EDuration="1.35202261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:10.347589138 +0000 UTC m=+1364.288137049" watchObservedRunningTime="2026-10-29 23:05:10.35202261 +0000 UTC m=+1364.292570519" Oct 29 23:05:10.632327 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:10.632297 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn" Oct 29 23:05:10.718447 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:10.718412 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g" Oct 29 23:05:10.758314 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:10.758242 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:05:10.760695 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:10.760666 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a84143c5-852c-4ebb-8aee-c5e3dd323ef6-kube-api-access-gtssv\" (UniqueName: \"kubernetes.io/projected/a84143c5-852c-4ebb-8aee-c5e3dd323ef6-kube-api-access-gtssv\") pod \"a84143c5-852c-4ebb-8aee-c5e3dd323ef6\" (UID: \"a84143c5-852c-4ebb-8aee-c5e3dd323ef6\") " Oct 29 23:05:10.760808 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:10.760734 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1425a1f4-640b-4c45-bae9-7e0d4b66afcf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1425a1f4-640b-4c45-bae9-7e0d4b66afcf-plugins-config-volume\") pod \"1425a1f4-640b-4c45-bae9-7e0d4b66afcf\" (UID: \"1425a1f4-640b-4c45-bae9-7e0d4b66afcf\") " Oct 29 23:05:10.760808 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:10.760775 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a84143c5-852c-4ebb-8aee-c5e3dd323ef6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a84143c5-852c-4ebb-8aee-c5e3dd323ef6-plugins-config-volume\") pod \"a84143c5-852c-4ebb-8aee-c5e3dd323ef6\" (UID: \"a84143c5-852c-4ebb-8aee-c5e3dd323ef6\") " Oct 29 23:05:10.761012 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:10.760996 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1425a1f4-640b-4c45-bae9-7e0d4b66afcf-kube-api-access-cs5jz\" (UniqueName: \"kubernetes.io/projected/1425a1f4-640b-4c45-bae9-7e0d4b66afcf-kube-api-access-cs5jz\") pod \"1425a1f4-640b-4c45-bae9-7e0d4b66afcf\" (UID: \"1425a1f4-640b-4c45-bae9-7e0d4b66afcf\") " Oct 29 23:05:10.761141 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:10.761101 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1425a1f4-640b-4c45-bae9-7e0d4b66afcf-plugins-config-volume" pod "1425a1f4-640b-4c45-bae9-7e0d4b66afcf" (UID: "1425a1f4-640b-4c45-bae9-7e0d4b66afcf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:10.761141 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:10.761108 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a84143c5-852c-4ebb-8aee-c5e3dd323ef6-plugins-config-volume" pod "a84143c5-852c-4ebb-8aee-c5e3dd323ef6" (UID: "a84143c5-852c-4ebb-8aee-c5e3dd323ef6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:10.762094 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:10.762074 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a84143c5-852c-4ebb-8aee-c5e3dd323ef6-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:10.762094 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:10.762096 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1425a1f4-640b-4c45-bae9-7e0d4b66afcf-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:10.764169 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:10.764145 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a84143c5-852c-4ebb-8aee-c5e3dd323ef6-kube-api-access-gtssv" pod "a84143c5-852c-4ebb-8aee-c5e3dd323ef6" (UID: "a84143c5-852c-4ebb-8aee-c5e3dd323ef6"). InnerVolumeSpecName "kube-api-access-gtssv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:10.764411 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:10.764383 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1425a1f4-640b-4c45-bae9-7e0d4b66afcf-kube-api-access-cs5jz" pod "1425a1f4-640b-4c45-bae9-7e0d4b66afcf" (UID: "1425a1f4-640b-4c45-bae9-7e0d4b66afcf"). InnerVolumeSpecName "kube-api-access-cs5jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:10.864177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:10.864136 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtssv\" (UniqueName: \"kubernetes.io/projected/a84143c5-852c-4ebb-8aee-c5e3dd323ef6-kube-api-access-gtssv\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:10.864177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:10.864171 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs5jz\" (UniqueName: \"kubernetes.io/projected/1425a1f4-640b-4c45-bae9-7e0d4b66afcf-kube-api-access-cs5jz\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.314729 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:11.314695 4821 generic.go:356] "Generic (PLEG): container finished" podID="a84143c5-852c-4ebb-8aee-c5e3dd323ef6" containerID="dde36c2a524b2e1da1ec2a6b280d5d00dec65e1bcc0adde2d9c159677a8df955" exitCode=0 Oct 29 23:05:11.315188 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:11.314736 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g" Oct 29 23:05:11.315188 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:11.314778 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g" event={"ID":"a84143c5-852c-4ebb-8aee-c5e3dd323ef6","Type":"ContainerDied","Data":"dde36c2a524b2e1da1ec2a6b280d5d00dec65e1bcc0adde2d9c159677a8df955"} Oct 29 23:05:11.315188 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:11.314829 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g" event={"ID":"a84143c5-852c-4ebb-8aee-c5e3dd323ef6","Type":"ContainerDied","Data":"08dbb5024f9d427a54dcbf77c44be5c84bd32e351e9023dfdaae17ba7d115407"} Oct 29 23:05:11.315188 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:11.314853 4821 scope.go:122] "RemoveContainer" containerID="dde36c2a524b2e1da1ec2a6b280d5d00dec65e1bcc0adde2d9c159677a8df955" Oct 29 23:05:11.316481 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:11.316456 4821 generic.go:356] "Generic (PLEG): container finished" podID="1425a1f4-640b-4c45-bae9-7e0d4b66afcf" containerID="a5133bbaeea0e3e64a6bf89d5f86db87684fc323dd3d4ca9c0c0544a27aa007a" exitCode=0 Oct 29 23:05:11.316591 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:11.316487 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn" event={"ID":"1425a1f4-640b-4c45-bae9-7e0d4b66afcf","Type":"ContainerDied","Data":"a5133bbaeea0e3e64a6bf89d5f86db87684fc323dd3d4ca9c0c0544a27aa007a"} Oct 29 23:05:11.316591 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:11.316541 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn" event={"ID":"1425a1f4-640b-4c45-bae9-7e0d4b66afcf","Type":"ContainerDied","Data":"54000fab6f816e127de11d87193a11a9f9a3ef3d53eaa52681f73a32753d3a75"} Oct 29 23:05:11.316591 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:11.316499 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn" Oct 29 23:05:11.326640 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:11.326620 4821 scope.go:122] "RemoveContainer" containerID="dde36c2a524b2e1da1ec2a6b280d5d00dec65e1bcc0adde2d9c159677a8df955" Oct 29 23:05:11.326947 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:05:11.326930 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dde36c2a524b2e1da1ec2a6b280d5d00dec65e1bcc0adde2d9c159677a8df955\": container with ID starting with dde36c2a524b2e1da1ec2a6b280d5d00dec65e1bcc0adde2d9c159677a8df955 not found: ID does not exist" containerID="dde36c2a524b2e1da1ec2a6b280d5d00dec65e1bcc0adde2d9c159677a8df955" Oct 29 23:05:11.327015 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:11.326955 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dde36c2a524b2e1da1ec2a6b280d5d00dec65e1bcc0adde2d9c159677a8df955"} err="failed to get container status \"dde36c2a524b2e1da1ec2a6b280d5d00dec65e1bcc0adde2d9c159677a8df955\": rpc error: code = NotFound desc = could not find container \"dde36c2a524b2e1da1ec2a6b280d5d00dec65e1bcc0adde2d9c159677a8df955\": container with ID starting with dde36c2a524b2e1da1ec2a6b280d5d00dec65e1bcc0adde2d9c159677a8df955 not found: ID does not exist" Oct 29 23:05:11.327015 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:11.326974 4821 scope.go:122] "RemoveContainer" containerID="a5133bbaeea0e3e64a6bf89d5f86db87684fc323dd3d4ca9c0c0544a27aa007a" Oct 29 23:05:11.336922 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:11.336905 4821 scope.go:122] "RemoveContainer" containerID="a5133bbaeea0e3e64a6bf89d5f86db87684fc323dd3d4ca9c0c0544a27aa007a" Oct 29 23:05:11.337188 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:05:11.337166 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5133bbaeea0e3e64a6bf89d5f86db87684fc323dd3d4ca9c0c0544a27aa007a\": container with ID starting with a5133bbaeea0e3e64a6bf89d5f86db87684fc323dd3d4ca9c0c0544a27aa007a not found: ID does not exist" containerID="a5133bbaeea0e3e64a6bf89d5f86db87684fc323dd3d4ca9c0c0544a27aa007a" Oct 29 23:05:11.337324 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:11.337192 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5133bbaeea0e3e64a6bf89d5f86db87684fc323dd3d4ca9c0c0544a27aa007a"} err="failed to get container status \"a5133bbaeea0e3e64a6bf89d5f86db87684fc323dd3d4ca9c0c0544a27aa007a\": rpc error: code = NotFound desc = could not find container \"a5133bbaeea0e3e64a6bf89d5f86db87684fc323dd3d4ca9c0c0544a27aa007a\": container with ID starting with a5133bbaeea0e3e64a6bf89d5f86db87684fc323dd3d4ca9c0c0544a27aa007a not found: ID does not exist" Oct 29 23:05:11.374349 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:11.374321 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g"] Oct 29 23:05:11.387940 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:11.387903 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-53a768-58fbd55ccf-vtr9g"] Oct 29 23:05:11.416143 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:11.416121 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn"] Oct 29 23:05:11.421587 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:11.421561 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-53a768-84f8cc9cf6-ltsvn"] Oct 29 23:05:12.175984 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:12.175952 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1425a1f4-640b-4c45-bae9-7e0d4b66afcf" path="/var/lib/kubelet/pods/1425a1f4-640b-4c45-bae9-7e0d4b66afcf/volumes" Oct 29 23:05:12.176565 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:12.176544 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a84143c5-852c-4ebb-8aee-c5e3dd323ef6" path="/var/lib/kubelet/pods/a84143c5-852c-4ebb-8aee-c5e3dd323ef6/volumes" Oct 29 23:05:12.738970 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:12.738942 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp" Oct 29 23:05:16.619644 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:16.619610 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx"] Oct 29 23:05:16.632683 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:16.632642 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-53a768-gqkxx"] Oct 29 23:05:18.171766 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:18.171725 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="150ec736-116e-4f11-b1be-b90c896896a4" path="/var/lib/kubelet/pods/150ec736-116e-4f11-b1be-b90c896896a4/volumes" Oct 29 23:05:19.758228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:19.758137 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:05:19.771096 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:19.771066 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:05:20.485319 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:20.485284 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:05:23.676288 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:23.665348 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs"] Oct 29 23:05:23.676288 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:23.665543 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" podUID="83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" containerName="istio-proxy" containerID="cri-o://979a10477c756ce5f6c1a1c6fbd1bc2b7616a3b5c12789e0f11ad78f78c36f3e" gracePeriod=30 Oct 29 23:05:23.923462 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:05:23.923421 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:05:24.260464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:24.260425 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97"] Oct 29 23:05:24.261151 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:24.261128 4821 cpu_manager.go:414] "containerMap: removing container" podUID="1425a1f4-640b-4c45-bae9-7e0d4b66afcf" containerName="payload-processing" Oct 29 23:05:24.261151 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:24.261150 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1425a1f4-640b-4c45-bae9-7e0d4b66afcf" containerName="payload-processing" Oct 29 23:05:24.261351 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:24.261187 4821 cpu_manager.go:414] "containerMap: removing container" podUID="a84143c5-852c-4ebb-8aee-c5e3dd323ef6" containerName="payload-pre-processing" Oct 29 23:05:24.261351 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:24.261193 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a84143c5-852c-4ebb-8aee-c5e3dd323ef6" containerName="payload-pre-processing" Oct 29 23:05:24.261351 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:24.261344 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="1425a1f4-640b-4c45-bae9-7e0d4b66afcf" containerName="payload-processing" Oct 29 23:05:24.261447 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:24.261358 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="a84143c5-852c-4ebb-8aee-c5e3dd323ef6" containerName="payload-pre-processing" Oct 29 23:05:24.265213 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:24.265184 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97" Oct 29 23:05:24.280322 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:24.280298 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97"] Oct 29 23:05:24.280743 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:24.280725 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:05:24.280795 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:24.280765 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:05:24.348651 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:24.348617 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fc7b828-c0f8-4dac-8424-0d4fe6100e62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97\" (UID: \"0fc7b828-c0f8-4dac-8424-0d4fe6100e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97" Oct 29 23:05:24.348833 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:24.348784 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fc7b828-c0f8-4dac-8424-0d4fe6100e62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97\" (UID: \"0fc7b828-c0f8-4dac-8424-0d4fe6100e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97" Oct 29 23:05:24.348883 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:24.348840 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sj5j\" (UniqueName: \"kubernetes.io/projected/0fc7b828-c0f8-4dac-8424-0d4fe6100e62-kube-api-access-8sj5j\") pod \"maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97\" (UID: \"0fc7b828-c0f8-4dac-8424-0d4fe6100e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97" Oct 29 23:05:24.450815 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:24.450780 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sj5j\" (UniqueName: \"kubernetes.io/projected/0fc7b828-c0f8-4dac-8424-0d4fe6100e62-kube-api-access-8sj5j\") pod \"maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97\" (UID: \"0fc7b828-c0f8-4dac-8424-0d4fe6100e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97" Oct 29 23:05:24.450995 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:24.450940 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fc7b828-c0f8-4dac-8424-0d4fe6100e62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97\" (UID: \"0fc7b828-c0f8-4dac-8424-0d4fe6100e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97" Oct 29 23:05:24.451055 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:24.451038 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fc7b828-c0f8-4dac-8424-0d4fe6100e62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97\" (UID: \"0fc7b828-c0f8-4dac-8424-0d4fe6100e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97" Oct 29 23:05:24.451658 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:24.451622 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fc7b828-c0f8-4dac-8424-0d4fe6100e62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97\" (UID: \"0fc7b828-c0f8-4dac-8424-0d4fe6100e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97" Oct 29 23:05:24.453776 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:24.453756 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fc7b828-c0f8-4dac-8424-0d4fe6100e62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97\" (UID: \"0fc7b828-c0f8-4dac-8424-0d4fe6100e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97" Oct 29 23:05:24.475432 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:24.475403 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sj5j\" (UniqueName: \"kubernetes.io/projected/0fc7b828-c0f8-4dac-8424-0d4fe6100e62-kube-api-access-8sj5j\") pod \"maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97\" (UID: \"0fc7b828-c0f8-4dac-8424-0d4fe6100e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97" Oct 29 23:05:24.580980 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:24.580895 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97" Oct 29 23:05:24.889518 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:24.889488 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97"] Oct 29 23:05:24.890355 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:05:24.890329 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc7b828_c0f8_4dac_8424_0d4fe6100e62.slice/crio-fb766224fa65a53ab12c489863a2d7fc0f4d4f45362bd5751eeb40089255b3bb WatchSource:0}: Error finding container fb766224fa65a53ab12c489863a2d7fc0f4d4f45362bd5751eeb40089255b3bb: Status 404 returned error can't find the container with id fb766224fa65a53ab12c489863a2d7fc0f4d4f45362bd5751eeb40089255b3bb Oct 29 23:05:25.526270 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:25.526237 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97" event={"ID":"0fc7b828-c0f8-4dac-8424-0d4fe6100e62","Type":"ContainerStarted","Data":"9c91e2c67cc9ee56bf23b73dd765ea7ced96a8a95f6ebeb03c533ac9f6128b82"} Oct 29 23:05:25.526463 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:25.526292 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97" event={"ID":"0fc7b828-c0f8-4dac-8424-0d4fe6100e62","Type":"ContainerStarted","Data":"fb766224fa65a53ab12c489863a2d7fc0f4d4f45362bd5751eeb40089255b3bb"} Oct 29 23:05:25.569395 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:25.569345 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97" podStartSLOduration=1.569332133 podStartE2EDuration="1.569332133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:25.567026607 +0000 UTC m=+1379.507574518" watchObservedRunningTime="2026-10-29 23:05:25.569332133 +0000 UTC m=+1379.509880042" Oct 29 23:05:26.537031 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:26.536998 4821 generic.go:356] "Generic (PLEG): container finished" podID="0fc7b828-c0f8-4dac-8424-0d4fe6100e62" containerID="9c91e2c67cc9ee56bf23b73dd765ea7ced96a8a95f6ebeb03c533ac9f6128b82" exitCode=0 Oct 29 23:05:26.537446 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:26.537095 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97" event={"ID":"0fc7b828-c0f8-4dac-8424-0d4fe6100e62","Type":"ContainerDied","Data":"9c91e2c67cc9ee56bf23b73dd765ea7ced96a8a95f6ebeb03c533ac9f6128b82"} Oct 29 23:05:27.964447 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:27.964420 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97" Oct 29 23:05:28.037492 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:28.037451 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fc7b828-c0f8-4dac-8424-0d4fe6100e62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fc7b828-c0f8-4dac-8424-0d4fe6100e62-maas-api-openshift-ca\") pod \"0fc7b828-c0f8-4dac-8424-0d4fe6100e62\" (UID: \"0fc7b828-c0f8-4dac-8424-0d4fe6100e62\") " Oct 29 23:05:28.037686 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:28.037528 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fc7b828-c0f8-4dac-8424-0d4fe6100e62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fc7b828-c0f8-4dac-8424-0d4fe6100e62-maas-api-xks-ca\") pod \"0fc7b828-c0f8-4dac-8424-0d4fe6100e62\" (UID: \"0fc7b828-c0f8-4dac-8424-0d4fe6100e62\") " Oct 29 23:05:28.037686 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:28.037573 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc7b828-c0f8-4dac-8424-0d4fe6100e62-kube-api-access-8sj5j\" (UniqueName: \"kubernetes.io/projected/0fc7b828-c0f8-4dac-8424-0d4fe6100e62-kube-api-access-8sj5j\") pod \"0fc7b828-c0f8-4dac-8424-0d4fe6100e62\" (UID: \"0fc7b828-c0f8-4dac-8424-0d4fe6100e62\") " Oct 29 23:05:28.038049 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:28.038022 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fc7b828-c0f8-4dac-8424-0d4fe6100e62-maas-api-openshift-ca" pod "0fc7b828-c0f8-4dac-8424-0d4fe6100e62" (UID: "0fc7b828-c0f8-4dac-8424-0d4fe6100e62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:28.038468 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:28.038440 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fc7b828-c0f8-4dac-8424-0d4fe6100e62-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:28.040918 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:28.040888 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fc7b828-c0f8-4dac-8424-0d4fe6100e62-maas-api-xks-ca" pod "0fc7b828-c0f8-4dac-8424-0d4fe6100e62" (UID: "0fc7b828-c0f8-4dac-8424-0d4fe6100e62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:28.041356 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:28.041329 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc7b828-c0f8-4dac-8424-0d4fe6100e62-kube-api-access-8sj5j" pod "0fc7b828-c0f8-4dac-8424-0d4fe6100e62" (UID: "0fc7b828-c0f8-4dac-8424-0d4fe6100e62"). InnerVolumeSpecName "kube-api-access-8sj5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:28.140367 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:28.140336 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fc7b828-c0f8-4dac-8424-0d4fe6100e62-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:28.140367 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:28.140365 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sj5j\" (UniqueName: \"kubernetes.io/projected/0fc7b828-c0f8-4dac-8424-0d4fe6100e62-kube-api-access-8sj5j\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:28.618814 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:28.618787 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97" Oct 29 23:05:28.618987 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:28.618826 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97" event={"ID":"0fc7b828-c0f8-4dac-8424-0d4fe6100e62","Type":"ContainerDied","Data":"fb766224fa65a53ab12c489863a2d7fc0f4d4f45362bd5751eeb40089255b3bb"} Oct 29 23:05:28.618987 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:28.618874 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb766224fa65a53ab12c489863a2d7fc0f4d4f45362bd5751eeb40089255b3bb" Oct 29 23:05:28.991497 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:28.991425 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:05:29.067240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.060264 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istiod-ca-cert\") pod \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " Oct 29 23:05:29.067240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.060377 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-workload-certs\") pod \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " Oct 29 23:05:29.067240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.060409 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-data\") pod \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " Oct 29 23:05:29.067240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.060452 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-kube-api-access-vstzv\" (UniqueName: \"kubernetes.io/projected/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-kube-api-access-vstzv\") pod \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " Oct 29 23:05:29.067240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.060489 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-token\" (UniqueName: \"kubernetes.io/projected/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-token\") pod \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " Oct 29 23:05:29.067240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.060601 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-credential-socket\") pod \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " Oct 29 23:05:29.067240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.060725 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-workload-socket\") pod \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " Oct 29 23:05:29.067240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.060769 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-podinfo\") pod \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " Oct 29 23:05:29.067240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.060905 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istiod-ca-cert" pod "83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" (UID: "83a4cc0f-6bd5-4098-87aa-ff42db1fbc10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:29.067240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.060913 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-envoy\") pod \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\" (UID: \"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10\") " Oct 29 23:05:29.067240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.061889 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:29.067240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.062222 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-workload-socket" pod "83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" (UID: "83a4cc0f-6bd5-4098-87aa-ff42db1fbc10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:29.067240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.062419 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-credential-socket" pod "83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" (UID: "83a4cc0f-6bd5-4098-87aa-ff42db1fbc10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:29.067240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.062891 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-data" pod "83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" (UID: "83a4cc0f-6bd5-4098-87aa-ff42db1fbc10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:29.067240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.064508 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-workload-certs" pod "83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" (UID: "83a4cc0f-6bd5-4098-87aa-ff42db1fbc10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:29.079568 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.076741 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-podinfo" pod "83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" (UID: "83a4cc0f-6bd5-4098-87aa-ff42db1fbc10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:05:29.080947 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.080899 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-envoy" pod "83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" (UID: "83a4cc0f-6bd5-4098-87aa-ff42db1fbc10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:29.081448 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.081294 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-token" pod "83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" (UID: "83a4cc0f-6bd5-4098-87aa-ff42db1fbc10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:29.081448 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.081323 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-kube-api-access-vstzv" pod "83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" (UID: "83a4cc0f-6bd5-4098-87aa-ff42db1fbc10"). InnerVolumeSpecName "kube-api-access-vstzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:29.164135 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.164093 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:29.164135 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.164134 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:29.164520 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.164149 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:29.164520 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.164163 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:29.164520 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.164177 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:29.164520 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.164190 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vstzv\" (UniqueName: \"kubernetes.io/projected/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-kube-api-access-vstzv\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:29.164520 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.164474 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:29.164520 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.164499 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:29.630615 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.630577 4821 generic.go:356] "Generic (PLEG): container finished" podID="83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" containerID="979a10477c756ce5f6c1a1c6fbd1bc2b7616a3b5c12789e0f11ad78f78c36f3e" exitCode=0 Oct 29 23:05:29.630840 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.630714 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" event={"ID":"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10","Type":"ContainerDied","Data":"979a10477c756ce5f6c1a1c6fbd1bc2b7616a3b5c12789e0f11ad78f78c36f3e"} Oct 29 23:05:29.630840 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.630770 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" event={"ID":"83a4cc0f-6bd5-4098-87aa-ff42db1fbc10","Type":"ContainerDied","Data":"b332ea071d9999517a8ccbeea21117a5b4be12a778cabc1a484f76516341c568"} Oct 29 23:05:29.630840 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.630801 4821 scope.go:122] "RemoveContainer" containerID="979a10477c756ce5f6c1a1c6fbd1bc2b7616a3b5c12789e0f11ad78f78c36f3e" Oct 29 23:05:29.631105 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.631088 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs" Oct 29 23:05:29.646769 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.646749 4821 scope.go:122] "RemoveContainer" containerID="fc92a35acaaa8e1ede0400b1dbc674112d76faec8ed5c23a2b0eb72e0bba1893" Oct 29 23:05:29.659557 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.659536 4821 scope.go:122] "RemoveContainer" containerID="979a10477c756ce5f6c1a1c6fbd1bc2b7616a3b5c12789e0f11ad78f78c36f3e" Oct 29 23:05:29.659878 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:05:29.659851 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"979a10477c756ce5f6c1a1c6fbd1bc2b7616a3b5c12789e0f11ad78f78c36f3e\": container with ID starting with 979a10477c756ce5f6c1a1c6fbd1bc2b7616a3b5c12789e0f11ad78f78c36f3e not found: ID does not exist" containerID="979a10477c756ce5f6c1a1c6fbd1bc2b7616a3b5c12789e0f11ad78f78c36f3e" Oct 29 23:05:29.659953 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.659892 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"979a10477c756ce5f6c1a1c6fbd1bc2b7616a3b5c12789e0f11ad78f78c36f3e"} err="failed to get container status \"979a10477c756ce5f6c1a1c6fbd1bc2b7616a3b5c12789e0f11ad78f78c36f3e\": rpc error: code = NotFound desc = could not find container \"979a10477c756ce5f6c1a1c6fbd1bc2b7616a3b5c12789e0f11ad78f78c36f3e\": container with ID starting with 979a10477c756ce5f6c1a1c6fbd1bc2b7616a3b5c12789e0f11ad78f78c36f3e not found: ID does not exist" Oct 29 23:05:29.659953 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.659918 4821 scope.go:122] "RemoveContainer" containerID="fc92a35acaaa8e1ede0400b1dbc674112d76faec8ed5c23a2b0eb72e0bba1893" Oct 29 23:05:29.660252 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:05:29.660172 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc92a35acaaa8e1ede0400b1dbc674112d76faec8ed5c23a2b0eb72e0bba1893\": container with ID starting with fc92a35acaaa8e1ede0400b1dbc674112d76faec8ed5c23a2b0eb72e0bba1893 not found: ID does not exist" containerID="fc92a35acaaa8e1ede0400b1dbc674112d76faec8ed5c23a2b0eb72e0bba1893" Oct 29 23:05:29.660703 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.660265 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc92a35acaaa8e1ede0400b1dbc674112d76faec8ed5c23a2b0eb72e0bba1893"} err="failed to get container status \"fc92a35acaaa8e1ede0400b1dbc674112d76faec8ed5c23a2b0eb72e0bba1893\": rpc error: code = NotFound desc = could not find container \"fc92a35acaaa8e1ede0400b1dbc674112d76faec8ed5c23a2b0eb72e0bba1893\": container with ID starting with fc92a35acaaa8e1ede0400b1dbc674112d76faec8ed5c23a2b0eb72e0bba1893 not found: ID does not exist" Oct 29 23:05:29.702135 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.702105 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs"] Oct 29 23:05:29.717602 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.717568 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-53a768-openshift-default-5d4578dd9c-lxvvs"] Oct 29 23:05:29.921493 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.921398 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl"] Oct 29 23:05:29.923292 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.923263 4821 cpu_manager.go:414] "containerMap: removing container" podUID="0fc7b828-c0f8-4dac-8424-0d4fe6100e62" containerName="revoke-keys" Oct 29 23:05:29.923498 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.923479 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc7b828-c0f8-4dac-8424-0d4fe6100e62" containerName="revoke-keys" Oct 29 23:05:29.923754 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.923728 4821 cpu_manager.go:414] "containerMap: removing container" podUID="83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" containerName="istio-proxy" Oct 29 23:05:29.923823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.923760 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" containerName="istio-proxy" Oct 29 23:05:29.923823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.923803 4821 cpu_manager.go:414] "containerMap: removing container" podUID="83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" containerName="istio-proxy" Oct 29 23:05:29.923823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.923815 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" containerName="istio-proxy" Oct 29 23:05:29.924170 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.924151 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fc7b828-c0f8-4dac-8424-0d4fe6100e62" containerName="revoke-keys" Oct 29 23:05:29.924281 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.924186 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" containerName="istio-proxy" Oct 29 23:05:29.924281 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.924221 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" containerName="istio-proxy" Oct 29 23:05:29.932375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.932341 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl" Oct 29 23:05:29.949937 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.949903 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-3f7d4b-serving-cert\"" Oct 29 23:05:29.953850 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.951911 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl"] Oct 29 23:05:29.982858 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.978528 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b0089e6-032c-4d01-8557-987929bb2de7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl\" (UID: \"8b0089e6-032c-4d01-8557-987929bb2de7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl" Oct 29 23:05:29.982858 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.978684 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7gwk\" (UniqueName: \"kubernetes.io/projected/8b0089e6-032c-4d01-8557-987929bb2de7-kube-api-access-r7gwk\") pod \"maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl\" (UID: \"8b0089e6-032c-4d01-8557-987929bb2de7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl" Oct 29 23:05:29.982858 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.978733 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b0089e6-032c-4d01-8557-987929bb2de7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl\" (UID: \"8b0089e6-032c-4d01-8557-987929bb2de7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl" Oct 29 23:05:29.982858 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:29.978875 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b0089e6-032c-4d01-8557-987929bb2de7-metrics-certs\") pod \"maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl\" (UID: \"8b0089e6-032c-4d01-8557-987929bb2de7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl" Oct 29 23:05:30.080864 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:30.080824 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b0089e6-032c-4d01-8557-987929bb2de7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl\" (UID: \"8b0089e6-032c-4d01-8557-987929bb2de7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl" Oct 29 23:05:30.081294 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:30.080927 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7gwk\" (UniqueName: \"kubernetes.io/projected/8b0089e6-032c-4d01-8557-987929bb2de7-kube-api-access-r7gwk\") pod \"maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl\" (UID: \"8b0089e6-032c-4d01-8557-987929bb2de7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl" Oct 29 23:05:30.081294 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:30.080969 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b0089e6-032c-4d01-8557-987929bb2de7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl\" (UID: \"8b0089e6-032c-4d01-8557-987929bb2de7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl" Oct 29 23:05:30.081294 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:30.081047 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b0089e6-032c-4d01-8557-987929bb2de7-metrics-certs\") pod \"maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl\" (UID: \"8b0089e6-032c-4d01-8557-987929bb2de7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl" Oct 29 23:05:30.081795 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:30.081728 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b0089e6-032c-4d01-8557-987929bb2de7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl\" (UID: \"8b0089e6-032c-4d01-8557-987929bb2de7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl" Oct 29 23:05:30.083729 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:30.083705 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b0089e6-032c-4d01-8557-987929bb2de7-metrics-certs\") pod \"maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl\" (UID: \"8b0089e6-032c-4d01-8557-987929bb2de7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl" Oct 29 23:05:30.084073 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:30.084037 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b0089e6-032c-4d01-8557-987929bb2de7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl\" (UID: \"8b0089e6-032c-4d01-8557-987929bb2de7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl" Oct 29 23:05:30.105968 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:30.105939 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7gwk\" (UniqueName: \"kubernetes.io/projected/8b0089e6-032c-4d01-8557-987929bb2de7-kube-api-access-r7gwk\") pod \"maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl\" (UID: \"8b0089e6-032c-4d01-8557-987929bb2de7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl" Oct 29 23:05:30.218762 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:30.218672 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83a4cc0f-6bd5-4098-87aa-ff42db1fbc10" path="/var/lib/kubelet/pods/83a4cc0f-6bd5-4098-87aa-ff42db1fbc10/volumes" Oct 29 23:05:30.261540 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:30.261510 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl" Oct 29 23:05:30.775780 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:05:30.775748 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0089e6_032c_4d01_8557_987929bb2de7.slice/crio-25e1b58bc2a038e763ee4c5ab58eac3f52fa2e52f791ab462e05e809f1cd3973 WatchSource:0}: Error finding container 25e1b58bc2a038e763ee4c5ab58eac3f52fa2e52f791ab462e05e809f1cd3973: Status 404 returned error can't find the container with id 25e1b58bc2a038e763ee4c5ab58eac3f52fa2e52f791ab462e05e809f1cd3973 Oct 29 23:05:30.821438 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:30.815304 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl"] Oct 29 23:05:31.702898 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:31.697674 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl" event={"ID":"8b0089e6-032c-4d01-8557-987929bb2de7","Type":"ContainerStarted","Data":"3630394ae606a60d7ac713056d032a7cdf75e4f7ae0bf104fd19f23ec8812c2c"} Oct 29 23:05:31.702898 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:31.697812 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl" event={"ID":"8b0089e6-032c-4d01-8557-987929bb2de7","Type":"ContainerStarted","Data":"25e1b58bc2a038e763ee4c5ab58eac3f52fa2e52f791ab462e05e809f1cd3973"} Oct 29 23:05:31.770290 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:31.748276 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl" podStartSLOduration=2.277576131 podStartE2EDuration="2.748255773s" totalImagesPullingTime="470.679642ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:31.743694335 +0000 UTC m=+1385.684242298" watchObservedRunningTime="2026-10-29 23:05:31.748255773 +0000 UTC m=+1385.688803683" Oct 29 23:05:32.725257 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:32.724399 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl" Oct 29 23:05:38.739529 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:38.739441 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl" Oct 29 23:05:38.859145 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:38.859110 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn"] Oct 29 23:05:38.864614 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:38.864594 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn" Oct 29 23:05:38.880360 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:38.880318 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:05:38.880557 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:38.880457 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:05:38.884027 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:38.884000 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn"] Oct 29 23:05:38.976484 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:38.976437 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdnz8\" (UniqueName: \"kubernetes.io/projected/e15fee17-c158-491c-943d-8ae6ebdbc4d7-kube-api-access-qdnz8\") pod \"maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn\" (UID: \"e15fee17-c158-491c-943d-8ae6ebdbc4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn" Oct 29 23:05:38.980671 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:38.976832 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e15fee17-c158-491c-943d-8ae6ebdbc4d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn\" (UID: \"e15fee17-c158-491c-943d-8ae6ebdbc4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn" Oct 29 23:05:38.980671 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:38.977018 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e15fee17-c158-491c-943d-8ae6ebdbc4d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn\" (UID: \"e15fee17-c158-491c-943d-8ae6ebdbc4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn" Oct 29 23:05:39.078015 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:39.077930 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdnz8\" (UniqueName: \"kubernetes.io/projected/e15fee17-c158-491c-943d-8ae6ebdbc4d7-kube-api-access-qdnz8\") pod \"maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn\" (UID: \"e15fee17-c158-491c-943d-8ae6ebdbc4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn" Oct 29 23:05:39.078015 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:39.077993 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e15fee17-c158-491c-943d-8ae6ebdbc4d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn\" (UID: \"e15fee17-c158-491c-943d-8ae6ebdbc4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn" Oct 29 23:05:39.078265 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:39.078093 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e15fee17-c158-491c-943d-8ae6ebdbc4d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn\" (UID: \"e15fee17-c158-491c-943d-8ae6ebdbc4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn" Oct 29 23:05:39.078841 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:39.078815 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e15fee17-c158-491c-943d-8ae6ebdbc4d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn\" (UID: \"e15fee17-c158-491c-943d-8ae6ebdbc4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn" Oct 29 23:05:39.080678 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:39.080650 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e15fee17-c158-491c-943d-8ae6ebdbc4d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn\" (UID: \"e15fee17-c158-491c-943d-8ae6ebdbc4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn" Oct 29 23:05:39.102123 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:39.102097 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdnz8\" (UniqueName: \"kubernetes.io/projected/e15fee17-c158-491c-943d-8ae6ebdbc4d7-kube-api-access-qdnz8\") pod \"maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn\" (UID: \"e15fee17-c158-491c-943d-8ae6ebdbc4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn" Oct 29 23:05:39.179279 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:39.179194 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn" Oct 29 23:05:39.493296 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:39.493266 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn"] Oct 29 23:05:39.493618 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:05:39.493577 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15fee17_c158_491c_943d_8ae6ebdbc4d7.slice/crio-79605cc38db47045bf03ce16e2f03ffd62211bd9a1cdd5cdc3154410916ff2de WatchSource:0}: Error finding container 79605cc38db47045bf03ce16e2f03ffd62211bd9a1cdd5cdc3154410916ff2de: Status 404 returned error can't find the container with id 79605cc38db47045bf03ce16e2f03ffd62211bd9a1cdd5cdc3154410916ff2de Oct 29 23:05:39.789502 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:39.789404 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn" event={"ID":"e15fee17-c158-491c-943d-8ae6ebdbc4d7","Type":"ContainerStarted","Data":"79605cc38db47045bf03ce16e2f03ffd62211bd9a1cdd5cdc3154410916ff2de"} Oct 29 23:05:40.798809 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:40.798714 4821 generic.go:356] "Generic (PLEG): container finished" podID="e15fee17-c158-491c-943d-8ae6ebdbc4d7" containerID="b60e84b15826677e11a1f4e07365074ff7cca3fd4c8be1242585cb0c1d82d065" exitCode=0 Oct 29 23:05:40.799226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:40.798800 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn" event={"ID":"e15fee17-c158-491c-943d-8ae6ebdbc4d7","Type":"ContainerDied","Data":"b60e84b15826677e11a1f4e07365074ff7cca3fd4c8be1242585cb0c1d82d065"} Oct 29 23:05:41.992576 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:41.992541 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn" Oct 29 23:05:42.050835 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:42.050800 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e15fee17-c158-491c-943d-8ae6ebdbc4d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e15fee17-c158-491c-943d-8ae6ebdbc4d7-maas-api-openshift-ca\") pod \"e15fee17-c158-491c-943d-8ae6ebdbc4d7\" (UID: \"e15fee17-c158-491c-943d-8ae6ebdbc4d7\") " Oct 29 23:05:42.051018 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:42.050866 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e15fee17-c158-491c-943d-8ae6ebdbc4d7-kube-api-access-qdnz8\" (UniqueName: \"kubernetes.io/projected/e15fee17-c158-491c-943d-8ae6ebdbc4d7-kube-api-access-qdnz8\") pod \"e15fee17-c158-491c-943d-8ae6ebdbc4d7\" (UID: \"e15fee17-c158-491c-943d-8ae6ebdbc4d7\") " Oct 29 23:05:42.051018 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:42.050961 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e15fee17-c158-491c-943d-8ae6ebdbc4d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e15fee17-c158-491c-943d-8ae6ebdbc4d7-maas-api-xks-ca\") pod \"e15fee17-c158-491c-943d-8ae6ebdbc4d7\" (UID: \"e15fee17-c158-491c-943d-8ae6ebdbc4d7\") " Oct 29 23:05:42.051229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:42.051184 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e15fee17-c158-491c-943d-8ae6ebdbc4d7-maas-api-openshift-ca" pod "e15fee17-c158-491c-943d-8ae6ebdbc4d7" (UID: "e15fee17-c158-491c-943d-8ae6ebdbc4d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:42.051905 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:42.051880 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e15fee17-c158-491c-943d-8ae6ebdbc4d7-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:42.054108 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:42.054069 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e15fee17-c158-491c-943d-8ae6ebdbc4d7-kube-api-access-qdnz8" pod "e15fee17-c158-491c-943d-8ae6ebdbc4d7" (UID: "e15fee17-c158-491c-943d-8ae6ebdbc4d7"). InnerVolumeSpecName "kube-api-access-qdnz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:42.054260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:42.054159 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e15fee17-c158-491c-943d-8ae6ebdbc4d7-maas-api-xks-ca" pod "e15fee17-c158-491c-943d-8ae6ebdbc4d7" (UID: "e15fee17-c158-491c-943d-8ae6ebdbc4d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:42.153999 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:42.153954 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdnz8\" (UniqueName: \"kubernetes.io/projected/e15fee17-c158-491c-943d-8ae6ebdbc4d7-kube-api-access-qdnz8\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:42.153999 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:42.154000 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e15fee17-c158-491c-943d-8ae6ebdbc4d7-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:42.821579 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:42.821544 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn" Oct 29 23:05:42.821756 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:42.821576 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn" event={"ID":"e15fee17-c158-491c-943d-8ae6ebdbc4d7","Type":"ContainerDied","Data":"79605cc38db47045bf03ce16e2f03ffd62211bd9a1cdd5cdc3154410916ff2de"} Oct 29 23:05:42.821756 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:42.821628 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79605cc38db47045bf03ce16e2f03ffd62211bd9a1cdd5cdc3154410916ff2de" Oct 29 23:05:46.890858 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:46.890827 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988"] Oct 29 23:05:46.891555 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:46.891533 4821 cpu_manager.go:414] "containerMap: removing container" podUID="e15fee17-c158-491c-943d-8ae6ebdbc4d7" containerName="revoke-keys" Oct 29 23:05:46.891673 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:46.891556 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e15fee17-c158-491c-943d-8ae6ebdbc4d7" containerName="revoke-keys" Oct 29 23:05:46.891791 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:46.891776 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="e15fee17-c158-491c-943d-8ae6ebdbc4d7" containerName="revoke-keys" Oct 29 23:05:46.895930 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:46.895914 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:46.936390 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:46.936361 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988"] Oct 29 23:05:47.023991 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.023940 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.024290 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.024030 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.024290 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.024261 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.024437 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.024316 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.024482 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.024456 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.024607 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.024512 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6chk\" (UniqueName: \"kubernetes.io/projected/e5ffde39-a456-47e3-8fea-4ce3f51084b3-kube-api-access-k6chk\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.024607 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.024572 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.024757 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.024653 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.024757 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.024733 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.126146 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.126111 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.126370 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.126178 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.126370 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.126242 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.126370 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.126293 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.126564 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.126381 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.126564 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.126420 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.126564 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.126473 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.126564 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.126523 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6chk\" (UniqueName: \"kubernetes.io/projected/e5ffde39-a456-47e3-8fea-4ce3f51084b3-kube-api-access-k6chk\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.126564 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.126551 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.126897 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.126619 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-data\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.126897 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.126708 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-credential-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.127176 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.126948 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-workload-socket\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.127176 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.126995 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istiod-ca-cert\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.127176 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.127057 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-workload-certs\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.128777 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.128750 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-envoy\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.128972 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.128955 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-podinfo\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.154071 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.154007 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6chk\" (UniqueName: \"kubernetes.io/projected/e5ffde39-a456-47e3-8fea-4ce3f51084b3-kube-api-access-k6chk\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.155292 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.155270 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-token\") pod \"e2e-worker-w1-e131af-openshift-default-c476c699c-qp988\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.209640 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.209614 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:47.307140 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.307108 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f"] Oct 29 23:05:47.311730 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.311711 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f" Oct 29 23:05:47.329613 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.328540 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-77238f-dockercfg-255x6\"" Oct 29 23:05:47.331353 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.331323 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-77238f\"" Oct 29 23:05:47.346895 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.346679 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f"] Oct 29 23:05:47.422174 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.422090 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs"] Oct 29 23:05:47.461687 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.459694 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs" Oct 29 23:05:47.478260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.465243 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59da5ac0-2bf1-4405-8a59-35c74f5fc209-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f\" (UID: \"59da5ac0-2bf1-4405-8a59-35c74f5fc209\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f" Oct 29 23:05:47.478260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.465564 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs\" (UID: \"c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs" Oct 29 23:05:47.478260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.465596 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prxzf\" (UniqueName: \"kubernetes.io/projected/59da5ac0-2bf1-4405-8a59-35c74f5fc209-kube-api-access-prxzf\") pod \"payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f\" (UID: \"59da5ac0-2bf1-4405-8a59-35c74f5fc209\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f" Oct 29 23:05:47.478260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.465634 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hxfl\" (UniqueName: \"kubernetes.io/projected/c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6-kube-api-access-2hxfl\") pod \"payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs\" (UID: \"c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs" Oct 29 23:05:47.478260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.466267 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs"] Oct 29 23:05:47.558016 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:05:47.557970 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ffde39_a456_47e3_8fea_4ce3f51084b3.slice/crio-2eb06ea3d2bd325694496d596293469a9039d4ba5f8fbf776172b35ec7ed6074 WatchSource:0}: Error finding container 2eb06ea3d2bd325694496d596293469a9039d4ba5f8fbf776172b35ec7ed6074: Status 404 returned error can't find the container with id 2eb06ea3d2bd325694496d596293469a9039d4ba5f8fbf776172b35ec7ed6074 Oct 29 23:05:47.563580 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.558397 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988"] Oct 29 23:05:47.570538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.570426 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59da5ac0-2bf1-4405-8a59-35c74f5fc209-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f\" (UID: \"59da5ac0-2bf1-4405-8a59-35c74f5fc209\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f" Oct 29 23:05:47.571478 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.571288 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:05:47.571621 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.571486 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs\" (UID: \"c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs" Oct 29 23:05:47.571621 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.571523 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prxzf\" (UniqueName: \"kubernetes.io/projected/59da5ac0-2bf1-4405-8a59-35c74f5fc209-kube-api-access-prxzf\") pod \"payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f\" (UID: \"59da5ac0-2bf1-4405-8a59-35c74f5fc209\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f" Oct 29 23:05:47.571621 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.571560 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hxfl\" (UniqueName: \"kubernetes.io/projected/c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6-kube-api-access-2hxfl\") pod \"payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs\" (UID: \"c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs" Oct 29 23:05:47.572302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.571240 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59da5ac0-2bf1-4405-8a59-35c74f5fc209-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f\" (UID: \"59da5ac0-2bf1-4405-8a59-35c74f5fc209\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f" Oct 29 23:05:47.572897 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.572686 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:05:47.573253 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.572842 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs\" (UID: \"c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs" Oct 29 23:05:47.573475 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.573427 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:05:47.646436 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.646377 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hxfl\" (UniqueName: \"kubernetes.io/projected/c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6-kube-api-access-2hxfl\") pod \"payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs\" (UID: \"c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs" Oct 29 23:05:47.646933 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.646889 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prxzf\" (UniqueName: \"kubernetes.io/projected/59da5ac0-2bf1-4405-8a59-35c74f5fc209-kube-api-access-prxzf\") pod \"payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f\" (UID: \"59da5ac0-2bf1-4405-8a59-35c74f5fc209\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f" Oct 29 23:05:47.784561 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.784532 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs" Oct 29 23:05:47.890247 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.887152 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" event={"ID":"e5ffde39-a456-47e3-8fea-4ce3f51084b3","Type":"ContainerStarted","Data":"a918a606c674fddd9381cd22be767cece4f4cc31b42ccb7914b0704f967612d5"} Oct 29 23:05:47.890247 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.887244 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" event={"ID":"e5ffde39-a456-47e3-8fea-4ce3f51084b3","Type":"ContainerStarted","Data":"2eb06ea3d2bd325694496d596293469a9039d4ba5f8fbf776172b35ec7ed6074"} Oct 29 23:05:47.931478 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.931431 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f" Oct 29 23:05:47.938666 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:47.938614 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" podStartSLOduration=1.938594196 podStartE2EDuration="1.938594196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:47.93371561 +0000 UTC m=+1401.874263520" watchObservedRunningTime="2026-10-29 23:05:47.938594196 +0000 UTC m=+1401.879142106" Oct 29 23:05:48.105771 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:48.105743 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs"] Oct 29 23:05:48.106160 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:05:48.106133 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56e5474_3d55_4c4a_a3f9_c137f7c2d0d6.slice/crio-fb512cd031f8ec3dad6872c848463d26f51b2e5b623a017614b7dc6ea711096a WatchSource:0}: Error finding container fb512cd031f8ec3dad6872c848463d26f51b2e5b623a017614b7dc6ea711096a: Status 404 returned error can't find the container with id fb512cd031f8ec3dad6872c848463d26f51b2e5b623a017614b7dc6ea711096a Oct 29 23:05:48.210583 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:48.210562 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:48.212490 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:48.212455 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 29 23:05:48.212612 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:48.212513 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" podUID="e5ffde39-a456-47e3-8fea-4ce3f51084b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 29 23:05:48.256951 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:05:48.256924 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59da5ac0_2bf1_4405_8a59_35c74f5fc209.slice/crio-253cced11d064723fe04f761207b0acfc78876fe8f0f43df67dd7450ad52a888 WatchSource:0}: Error finding container 253cced11d064723fe04f761207b0acfc78876fe8f0f43df67dd7450ad52a888: Status 404 returned error can't find the container with id 253cced11d064723fe04f761207b0acfc78876fe8f0f43df67dd7450ad52a888 Oct 29 23:05:48.261598 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:48.261571 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f"] Oct 29 23:05:48.896837 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:48.896794 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f" event={"ID":"59da5ac0-2bf1-4405-8a59-35c74f5fc209","Type":"ContainerStarted","Data":"fc596080f0f369ab3f468e26691cd3f6688d0fb175d153a036f3862ae89373dc"} Oct 29 23:05:48.897034 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:48.896867 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f" Oct 29 23:05:48.897034 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:48.896893 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f" event={"ID":"59da5ac0-2bf1-4405-8a59-35c74f5fc209","Type":"ContainerStarted","Data":"253cced11d064723fe04f761207b0acfc78876fe8f0f43df67dd7450ad52a888"} Oct 29 23:05:48.898698 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:48.898666 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs" event={"ID":"c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6","Type":"ContainerStarted","Data":"9f2bf3c8769f0d7402349032a98d7f42b540ae0513eff052f637b5885b50b50e"} Oct 29 23:05:48.898825 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:48.898720 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs" event={"ID":"c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6","Type":"ContainerStarted","Data":"fb512cd031f8ec3dad6872c848463d26f51b2e5b623a017614b7dc6ea711096a"} Oct 29 23:05:48.898825 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:48.898751 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs" Oct 29 23:05:48.945555 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:48.945502 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f" podStartSLOduration=1.945487543 podStartE2EDuration="1.945487543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:48.941518577 +0000 UTC m=+1402.882066490" watchObservedRunningTime="2026-10-29 23:05:48.945487543 +0000 UTC m=+1402.886035453" Oct 29 23:05:48.989235 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:48.989148 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs" podStartSLOduration=1.989133774 podStartE2EDuration="1.989133774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:48.987776594 +0000 UTC m=+1402.928324506" watchObservedRunningTime="2026-10-29 23:05:48.989133774 +0000 UTC m=+1402.929681685" Oct 29 23:05:49.210486 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:49.210374 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 29 23:05:49.210486 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:49.210474 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" podUID="e5ffde39-a456-47e3-8fea-4ce3f51084b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 29 23:05:50.210651 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:50.210618 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 29 23:05:50.211028 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:50.210676 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" podUID="e5ffde39-a456-47e3-8fea-4ce3f51084b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 29 23:05:51.218227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:51.211711 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 29 23:05:51.218227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:51.211777 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" podUID="e5ffde39-a456-47e3-8fea-4ce3f51084b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 29 23:05:51.912718 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:51.912682 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97"] Oct 29 23:05:51.953968 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:51.953935 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c73f1d-krj97"] Oct 29 23:05:52.173007 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:52.172924 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fc7b828-c0f8-4dac-8424-0d4fe6100e62" path="/var/lib/kubelet/pods/0fc7b828-c0f8-4dac-8424-0d4fe6100e62/volumes" Oct 29 23:05:52.210506 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:52.210476 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 29 23:05:52.210691 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:52.210529 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" podUID="e5ffde39-a456-47e3-8fea-4ce3f51084b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 29 23:05:53.045232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:53.033376 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7"] Oct 29 23:05:53.045232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:53.033619 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" podUID="5010e064-9f9b-4de0-8b7b-4c132c3f5026" containerName="istio-proxy" containerID="cri-o://4b31acace189057cb226559e2c87f84d1a6f46983a9dbce039d28fdc0d742de2" gracePeriod=30 Oct 29 23:05:53.210673 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:53.210630 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 29 23:05:53.210857 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:53.210700 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" podUID="e5ffde39-a456-47e3-8fea-4ce3f51084b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 29 23:05:53.980177 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:05:53.980134 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:05:54.209945 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:54.209907 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 29 23:05:54.210375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:54.209963 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" podUID="e5ffde39-a456-47e3-8fea-4ce3f51084b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 29 23:05:55.418935 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:55.418893 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:55.419472 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:55.419379 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:55.420440 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:55.420417 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:05:56.215813 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:56.215779 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw"] Oct 29 23:05:56.216030 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:56.215986 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" podUID="c103357b-acd8-44e6-9600-c56965634907" containerName="istio-proxy" containerID="cri-o://6f6835c53e154f1a4f6b04702bc55223cf26d5db328734a4cebd949b9cbf9be6" gracePeriod=30 Oct 29 23:05:58.766071 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.766037 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 23:05:58.887369 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.887329 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-podinfo\") pod \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " Oct 29 23:05:58.887578 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.887393 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5010e064-9f9b-4de0-8b7b-4c132c3f5026-kube-api-access-hmppg\" (UniqueName: \"kubernetes.io/projected/5010e064-9f9b-4de0-8b7b-4c132c3f5026-kube-api-access-hmppg\") pod \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " Oct 29 23:05:58.887578 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.887425 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-token\" (UniqueName: \"kubernetes.io/projected/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-token\") pod \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " Oct 29 23:05:58.887578 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.887481 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-workload-certs\") pod \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " Oct 29 23:05:58.887578 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.887552 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-credential-socket\") pod \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " Oct 29 23:05:58.887822 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.887613 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-envoy\") pod \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " Oct 29 23:05:58.887822 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.887643 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istiod-ca-cert\") pod \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " Oct 29 23:05:58.887822 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.887672 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-data\") pod \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " Oct 29 23:05:58.887822 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.887717 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-workload-socket\") pod \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\" (UID: \"5010e064-9f9b-4de0-8b7b-4c132c3f5026\") " Oct 29 23:05:58.888031 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.887829 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-workload-certs" pod "5010e064-9f9b-4de0-8b7b-4c132c3f5026" (UID: "5010e064-9f9b-4de0-8b7b-4c132c3f5026"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:58.888031 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.887850 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-credential-socket" pod "5010e064-9f9b-4de0-8b7b-4c132c3f5026" (UID: "5010e064-9f9b-4de0-8b7b-4c132c3f5026"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:58.888124 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.888038 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-data" pod "5010e064-9f9b-4de0-8b7b-4c132c3f5026" (UID: "5010e064-9f9b-4de0-8b7b-4c132c3f5026"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:58.888178 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.888158 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-workload-socket" pod "5010e064-9f9b-4de0-8b7b-4c132c3f5026" (UID: "5010e064-9f9b-4de0-8b7b-4c132c3f5026"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:58.888429 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.888402 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istiod-ca-cert" pod "5010e064-9f9b-4de0-8b7b-4c132c3f5026" (UID: "5010e064-9f9b-4de0-8b7b-4c132c3f5026"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:58.888623 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.888604 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:58.888713 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.888629 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:58.888713 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.888646 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:58.888713 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.888659 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:58.888713 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.888672 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:58.890663 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.890627 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-podinfo" pod "5010e064-9f9b-4de0-8b7b-4c132c3f5026" (UID: "5010e064-9f9b-4de0-8b7b-4c132c3f5026"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:05:58.890930 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.890910 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5010e064-9f9b-4de0-8b7b-4c132c3f5026-kube-api-access-hmppg" pod "5010e064-9f9b-4de0-8b7b-4c132c3f5026" (UID: "5010e064-9f9b-4de0-8b7b-4c132c3f5026"). InnerVolumeSpecName "kube-api-access-hmppg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:58.891044 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.891029 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-token" pod "5010e064-9f9b-4de0-8b7b-4c132c3f5026" (UID: "5010e064-9f9b-4de0-8b7b-4c132c3f5026"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:58.891309 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.891290 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-envoy" pod "5010e064-9f9b-4de0-8b7b-4c132c3f5026" (UID: "5010e064-9f9b-4de0-8b7b-4c132c3f5026"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:58.990148 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.990112 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:58.990148 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.990143 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmppg\" (UniqueName: \"kubernetes.io/projected/5010e064-9f9b-4de0-8b7b-4c132c3f5026-kube-api-access-hmppg\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:58.990148 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.990154 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:58.990407 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:58.990163 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5010e064-9f9b-4de0-8b7b-4c132c3f5026-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:05:59.028322 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:59.028284 4821 generic.go:356] "Generic (PLEG): container finished" podID="5010e064-9f9b-4de0-8b7b-4c132c3f5026" containerID="4b31acace189057cb226559e2c87f84d1a6f46983a9dbce039d28fdc0d742de2" exitCode=0 Oct 29 23:05:59.028545 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:59.028342 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" Oct 29 23:05:59.028545 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:59.028378 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" event={"ID":"5010e064-9f9b-4de0-8b7b-4c132c3f5026","Type":"ContainerDied","Data":"4b31acace189057cb226559e2c87f84d1a6f46983a9dbce039d28fdc0d742de2"} Oct 29 23:05:59.028545 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:59.028448 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7" event={"ID":"5010e064-9f9b-4de0-8b7b-4c132c3f5026","Type":"ContainerDied","Data":"e97b2cba4e93c50edb1a5283d9da052cc99cbe6064e0deff4a1354de8376a9cd"} Oct 29 23:05:59.028545 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:59.028480 4821 scope.go:122] "RemoveContainer" containerID="4b31acace189057cb226559e2c87f84d1a6f46983a9dbce039d28fdc0d742de2" Oct 29 23:05:59.051010 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:59.050846 4821 scope.go:122] "RemoveContainer" containerID="4b31acace189057cb226559e2c87f84d1a6f46983a9dbce039d28fdc0d742de2" Oct 29 23:05:59.052626 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:05:59.052529 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b31acace189057cb226559e2c87f84d1a6f46983a9dbce039d28fdc0d742de2\": container with ID starting with 4b31acace189057cb226559e2c87f84d1a6f46983a9dbce039d28fdc0d742de2 not found: ID does not exist" containerID="4b31acace189057cb226559e2c87f84d1a6f46983a9dbce039d28fdc0d742de2" Oct 29 23:05:59.052626 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:59.052570 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b31acace189057cb226559e2c87f84d1a6f46983a9dbce039d28fdc0d742de2"} err="failed to get container status \"4b31acace189057cb226559e2c87f84d1a6f46983a9dbce039d28fdc0d742de2\": rpc error: code = NotFound desc = could not find container \"4b31acace189057cb226559e2c87f84d1a6f46983a9dbce039d28fdc0d742de2\": container with ID starting with 4b31acace189057cb226559e2c87f84d1a6f46983a9dbce039d28fdc0d742de2 not found: ID does not exist" Oct 29 23:05:59.112224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:59.110966 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7"] Oct 29 23:05:59.140319 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:59.128590 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-fbql7"] Oct 29 23:05:59.910130 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:59.910098 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f" Oct 29 23:05:59.910582 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:05:59.910178 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs" Oct 29 23:06:00.171877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:00.171803 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5010e064-9f9b-4de0-8b7b-4c132c3f5026" path="/var/lib/kubelet/pods/5010e064-9f9b-4de0-8b7b-4c132c3f5026/volumes" Oct 29 23:06:00.836873 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:00.836837 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw"] Oct 29 23:06:00.837460 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:00.837445 4821 cpu_manager.go:414] "containerMap: removing container" podUID="5010e064-9f9b-4de0-8b7b-4c132c3f5026" containerName="istio-proxy" Oct 29 23:06:00.837517 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:00.837461 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5010e064-9f9b-4de0-8b7b-4c132c3f5026" containerName="istio-proxy" Oct 29 23:06:00.837626 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:00.837616 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="5010e064-9f9b-4de0-8b7b-4c132c3f5026" containerName="istio-proxy" Oct 29 23:06:00.842711 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:00.842691 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw" Oct 29 23:06:00.868861 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:00.868821 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw"] Oct 29 23:06:00.903645 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:00.903608 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:06:00.905191 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:00.905168 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:06:01.032235 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.032158 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f8724b-cf95-4a2d-8061-cb9236f16c8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw\" (UID: \"f0f8724b-cf95-4a2d-8061-cb9236f16c8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw" Oct 29 23:06:01.032645 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.032312 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmlzp\" (UniqueName: \"kubernetes.io/projected/f0f8724b-cf95-4a2d-8061-cb9236f16c8f-kube-api-access-cmlzp\") pod \"maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw\" (UID: \"f0f8724b-cf95-4a2d-8061-cb9236f16c8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw" Oct 29 23:06:01.032645 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.032460 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f8724b-cf95-4a2d-8061-cb9236f16c8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw\" (UID: \"f0f8724b-cf95-4a2d-8061-cb9236f16c8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw" Oct 29 23:06:01.134374 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.134343 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f8724b-cf95-4a2d-8061-cb9236f16c8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw\" (UID: \"f0f8724b-cf95-4a2d-8061-cb9236f16c8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw" Oct 29 23:06:01.134583 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.134407 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmlzp\" (UniqueName: \"kubernetes.io/projected/f0f8724b-cf95-4a2d-8061-cb9236f16c8f-kube-api-access-cmlzp\") pod \"maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw\" (UID: \"f0f8724b-cf95-4a2d-8061-cb9236f16c8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw" Oct 29 23:06:01.134583 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.134494 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f8724b-cf95-4a2d-8061-cb9236f16c8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw\" (UID: \"f0f8724b-cf95-4a2d-8061-cb9236f16c8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw" Oct 29 23:06:01.135125 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.135100 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f8724b-cf95-4a2d-8061-cb9236f16c8f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw\" (UID: \"f0f8724b-cf95-4a2d-8061-cb9236f16c8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw" Oct 29 23:06:01.137283 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.137187 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f8724b-cf95-4a2d-8061-cb9236f16c8f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw\" (UID: \"f0f8724b-cf95-4a2d-8061-cb9236f16c8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw" Oct 29 23:06:01.159414 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.159391 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmlzp\" (UniqueName: \"kubernetes.io/projected/f0f8724b-cf95-4a2d-8061-cb9236f16c8f-kube-api-access-cmlzp\") pod \"maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw\" (UID: \"f0f8724b-cf95-4a2d-8061-cb9236f16c8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw" Oct 29 23:06:01.171339 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.171318 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw" Oct 29 23:06:01.487811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.487725 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw"] Oct 29 23:06:01.490056 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:06:01.490022 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f8724b_cf95_4a2d_8061_cb9236f16c8f.slice/crio-93dc0b58fbd9f63f8ea13b12bbd92b84135821a186aff194e244cc4ba6b3803b WatchSource:0}: Error finding container 93dc0b58fbd9f63f8ea13b12bbd92b84135821a186aff194e244cc4ba6b3803b: Status 404 returned error can't find the container with id 93dc0b58fbd9f63f8ea13b12bbd92b84135821a186aff194e244cc4ba6b3803b Oct 29 23:06:01.497277 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.497250 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 23:06:01.649263 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.649180 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-credential-socket\") pod \"c103357b-acd8-44e6-9600-c56965634907\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " Oct 29 23:06:01.649263 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.649257 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-istio-envoy\") pod \"c103357b-acd8-44e6-9600-c56965634907\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " Oct 29 23:06:01.649450 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.649295 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-workload-socket\") pod \"c103357b-acd8-44e6-9600-c56965634907\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " Oct 29 23:06:01.649450 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.649331 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-istio-data\") pod \"c103357b-acd8-44e6-9600-c56965634907\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " Oct 29 23:06:01.649538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.649465 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-credential-socket" pod "c103357b-acd8-44e6-9600-c56965634907" (UID: "c103357b-acd8-44e6-9600-c56965634907"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:01.649538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.649489 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c103357b-acd8-44e6-9600-c56965634907-istio-token\" (UniqueName: \"kubernetes.io/projected/c103357b-acd8-44e6-9600-c56965634907-istio-token\") pod \"c103357b-acd8-44e6-9600-c56965634907\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " Oct 29 23:06:01.649538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.649524 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c103357b-acd8-44e6-9600-c56965634907-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c103357b-acd8-44e6-9600-c56965634907-istiod-ca-cert\") pod \"c103357b-acd8-44e6-9600-c56965634907\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " Oct 29 23:06:01.649663 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.649567 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-workload-socket" pod "c103357b-acd8-44e6-9600-c56965634907" (UID: "c103357b-acd8-44e6-9600-c56965634907"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:01.649663 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.649584 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c103357b-acd8-44e6-9600-c56965634907-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c103357b-acd8-44e6-9600-c56965634907-istio-podinfo\") pod \"c103357b-acd8-44e6-9600-c56965634907\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " Oct 29 23:06:01.649663 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.649640 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-workload-certs\") pod \"c103357b-acd8-44e6-9600-c56965634907\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " Oct 29 23:06:01.649663 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.649648 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-istio-data" pod "c103357b-acd8-44e6-9600-c56965634907" (UID: "c103357b-acd8-44e6-9600-c56965634907"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:01.649939 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.649837 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c103357b-acd8-44e6-9600-c56965634907-kube-api-access-8vfdd\" (UniqueName: \"kubernetes.io/projected/c103357b-acd8-44e6-9600-c56965634907-kube-api-access-8vfdd\") pod \"c103357b-acd8-44e6-9600-c56965634907\" (UID: \"c103357b-acd8-44e6-9600-c56965634907\") " Oct 29 23:06:01.649939 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.649868 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c103357b-acd8-44e6-9600-c56965634907-istiod-ca-cert" pod "c103357b-acd8-44e6-9600-c56965634907" (UID: "c103357b-acd8-44e6-9600-c56965634907"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:01.650052 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.649995 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-workload-certs" pod "c103357b-acd8-44e6-9600-c56965634907" (UID: "c103357b-acd8-44e6-9600-c56965634907"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:01.650758 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.650731 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:01.650874 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.650763 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:01.650874 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.650780 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:01.650874 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.650793 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c103357b-acd8-44e6-9600-c56965634907-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:01.650874 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.650806 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:01.651942 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.651920 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c103357b-acd8-44e6-9600-c56965634907-istio-podinfo" pod "c103357b-acd8-44e6-9600-c56965634907" (UID: "c103357b-acd8-44e6-9600-c56965634907"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:06:01.652083 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.652066 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c103357b-acd8-44e6-9600-c56965634907-kube-api-access-8vfdd" pod "c103357b-acd8-44e6-9600-c56965634907" (UID: "c103357b-acd8-44e6-9600-c56965634907"). InnerVolumeSpecName "kube-api-access-8vfdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:01.652233 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.652194 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-istio-envoy" pod "c103357b-acd8-44e6-9600-c56965634907" (UID: "c103357b-acd8-44e6-9600-c56965634907"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:01.652336 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.652239 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c103357b-acd8-44e6-9600-c56965634907-istio-token" pod "c103357b-acd8-44e6-9600-c56965634907" (UID: "c103357b-acd8-44e6-9600-c56965634907"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:01.752694 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.752668 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c103357b-acd8-44e6-9600-c56965634907-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:01.752694 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.752692 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vfdd\" (UniqueName: \"kubernetes.io/projected/c103357b-acd8-44e6-9600-c56965634907-kube-api-access-8vfdd\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:01.752884 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.752704 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c103357b-acd8-44e6-9600-c56965634907-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:01.752884 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:01.752713 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c103357b-acd8-44e6-9600-c56965634907-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:02.058558 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.058472 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw" event={"ID":"f0f8724b-cf95-4a2d-8061-cb9236f16c8f","Type":"ContainerStarted","Data":"f099fba34c63e9a0c5c2973192d12abd60fb3e9116869e25153fe9cca1897db2"} Oct 29 23:06:02.058558 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.058527 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw" event={"ID":"f0f8724b-cf95-4a2d-8061-cb9236f16c8f","Type":"ContainerStarted","Data":"93dc0b58fbd9f63f8ea13b12bbd92b84135821a186aff194e244cc4ba6b3803b"} Oct 29 23:06:02.059914 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.059884 4821 generic.go:356] "Generic (PLEG): container finished" podID="c103357b-acd8-44e6-9600-c56965634907" containerID="6f6835c53e154f1a4f6b04702bc55223cf26d5db328734a4cebd949b9cbf9be6" exitCode=0 Oct 29 23:06:02.060065 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.059928 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" Oct 29 23:06:02.060065 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.059968 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" event={"ID":"c103357b-acd8-44e6-9600-c56965634907","Type":"ContainerDied","Data":"6f6835c53e154f1a4f6b04702bc55223cf26d5db328734a4cebd949b9cbf9be6"} Oct 29 23:06:02.060065 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.060025 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw" event={"ID":"c103357b-acd8-44e6-9600-c56965634907","Type":"ContainerDied","Data":"0905d89c49df2ed52a1db101d9dc4f5981a6cab39c5414043e1247f9706141f6"} Oct 29 23:06:02.060065 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.060053 4821 scope.go:122] "RemoveContainer" containerID="6f6835c53e154f1a4f6b04702bc55223cf26d5db328734a4cebd949b9cbf9be6" Oct 29 23:06:02.071255 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.071237 4821 scope.go:122] "RemoveContainer" containerID="6f6835c53e154f1a4f6b04702bc55223cf26d5db328734a4cebd949b9cbf9be6" Oct 29 23:06:02.071565 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:06:02.071545 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f6835c53e154f1a4f6b04702bc55223cf26d5db328734a4cebd949b9cbf9be6\": container with ID starting with 6f6835c53e154f1a4f6b04702bc55223cf26d5db328734a4cebd949b9cbf9be6 not found: ID does not exist" containerID="6f6835c53e154f1a4f6b04702bc55223cf26d5db328734a4cebd949b9cbf9be6" Oct 29 23:06:02.071618 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.071573 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f6835c53e154f1a4f6b04702bc55223cf26d5db328734a4cebd949b9cbf9be6"} err="failed to get container status \"6f6835c53e154f1a4f6b04702bc55223cf26d5db328734a4cebd949b9cbf9be6\": rpc error: code = NotFound desc = could not find container \"6f6835c53e154f1a4f6b04702bc55223cf26d5db328734a4cebd949b9cbf9be6\": container with ID starting with 6f6835c53e154f1a4f6b04702bc55223cf26d5db328734a4cebd949b9cbf9be6 not found: ID does not exist" Oct 29 23:06:02.105330 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.105282 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw" podStartSLOduration=2.105270012 podStartE2EDuration="2.105270012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:02.100978739 +0000 UTC m=+1416.041526648" watchObservedRunningTime="2026-10-29 23:06:02.105270012 +0000 UTC m=+1416.045817922" Oct 29 23:06:02.143784 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.143746 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw"] Oct 29 23:06:02.149417 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.149397 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-272xw"] Oct 29 23:06:02.171858 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.171832 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c103357b-acd8-44e6-9600-c56965634907" path="/var/lib/kubelet/pods/c103357b-acd8-44e6-9600-c56965634907/volumes" Oct 29 23:06:02.246386 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.246356 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa7f0ef4-s9tsc"] Oct 29 23:06:02.247300 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.247283 4821 cpu_manager.go:414] "containerMap: removing container" podUID="c103357b-acd8-44e6-9600-c56965634907" containerName="istio-proxy" Oct 29 23:06:02.247376 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.247303 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c103357b-acd8-44e6-9600-c56965634907" containerName="istio-proxy" Oct 29 23:06:02.247457 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.247445 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="c103357b-acd8-44e6-9600-c56965634907" containerName="istio-proxy" Oct 29 23:06:02.252845 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.252825 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f0ef4-s9tsc" Oct 29 23:06:02.258571 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.258547 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7a10320-b197-4471-ac63-fd5041d2a8d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa7f0ef4-s9tsc\" (UID: \"f7a10320-b197-4471-ac63-fd5041d2a8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f0ef4-s9tsc" Oct 29 23:06:02.258741 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.258710 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49gqq\" (UniqueName: \"kubernetes.io/projected/f7a10320-b197-4471-ac63-fd5041d2a8d4-kube-api-access-49gqq\") pod \"maas-api-revoke-sub-fa7f0ef4-s9tsc\" (UID: \"f7a10320-b197-4471-ac63-fd5041d2a8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f0ef4-s9tsc" Oct 29 23:06:02.258959 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.258923 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7a10320-b197-4471-ac63-fd5041d2a8d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa7f0ef4-s9tsc\" (UID: \"f7a10320-b197-4471-ac63-fd5041d2a8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f0ef4-s9tsc" Oct 29 23:06:02.271164 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.271137 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa7f0ef4-s9tsc"] Oct 29 23:06:02.360857 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.360825 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7a10320-b197-4471-ac63-fd5041d2a8d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa7f0ef4-s9tsc\" (UID: \"f7a10320-b197-4471-ac63-fd5041d2a8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f0ef4-s9tsc" Oct 29 23:06:02.361010 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.360943 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7a10320-b197-4471-ac63-fd5041d2a8d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa7f0ef4-s9tsc\" (UID: \"f7a10320-b197-4471-ac63-fd5041d2a8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f0ef4-s9tsc" Oct 29 23:06:02.361066 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.361027 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49gqq\" (UniqueName: \"kubernetes.io/projected/f7a10320-b197-4471-ac63-fd5041d2a8d4-kube-api-access-49gqq\") pod \"maas-api-revoke-sub-fa7f0ef4-s9tsc\" (UID: \"f7a10320-b197-4471-ac63-fd5041d2a8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f0ef4-s9tsc" Oct 29 23:06:02.361676 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.361592 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7a10320-b197-4471-ac63-fd5041d2a8d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa7f0ef4-s9tsc\" (UID: \"f7a10320-b197-4471-ac63-fd5041d2a8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f0ef4-s9tsc" Oct 29 23:06:02.363396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.363374 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7a10320-b197-4471-ac63-fd5041d2a8d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa7f0ef4-s9tsc\" (UID: \"f7a10320-b197-4471-ac63-fd5041d2a8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f0ef4-s9tsc" Oct 29 23:06:02.385782 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.385752 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49gqq\" (UniqueName: \"kubernetes.io/projected/f7a10320-b197-4471-ac63-fd5041d2a8d4-kube-api-access-49gqq\") pod \"maas-api-revoke-sub-fa7f0ef4-s9tsc\" (UID: \"f7a10320-b197-4471-ac63-fd5041d2a8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f0ef4-s9tsc" Oct 29 23:06:02.572914 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.572884 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f0ef4-s9tsc" Oct 29 23:06:02.886722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:02.886694 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa7f0ef4-s9tsc"] Oct 29 23:06:02.887360 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:06:02.887334 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a10320_b197_4471_ac63_fd5041d2a8d4.slice/crio-587606cd021ff4ad4661a05514c6045d5d28aaf8dfdf36220e08d15b6024b37c WatchSource:0}: Error finding container 587606cd021ff4ad4661a05514c6045d5d28aaf8dfdf36220e08d15b6024b37c: Status 404 returned error can't find the container with id 587606cd021ff4ad4661a05514c6045d5d28aaf8dfdf36220e08d15b6024b37c Oct 29 23:06:03.071024 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:03.070987 4821 generic.go:356] "Generic (PLEG): container finished" podID="f0f8724b-cf95-4a2d-8061-cb9236f16c8f" containerID="f099fba34c63e9a0c5c2973192d12abd60fb3e9116869e25153fe9cca1897db2" exitCode=0 Oct 29 23:06:03.071536 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:03.071031 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw" event={"ID":"f0f8724b-cf95-4a2d-8061-cb9236f16c8f","Type":"ContainerDied","Data":"f099fba34c63e9a0c5c2973192d12abd60fb3e9116869e25153fe9cca1897db2"} Oct 29 23:06:03.074118 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:03.074094 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f0ef4-s9tsc" event={"ID":"f7a10320-b197-4471-ac63-fd5041d2a8d4","Type":"ContainerStarted","Data":"587606cd021ff4ad4661a05514c6045d5d28aaf8dfdf36220e08d15b6024b37c"} Oct 29 23:06:04.086731 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:04.086604 4821 generic.go:356] "Generic (PLEG): container finished" podID="f7a10320-b197-4471-ac63-fd5041d2a8d4" containerID="344e8353e6748bc1d43ea4c3554307d6c38d222ea96c756027bfb940c6b432b1" exitCode=0 Oct 29 23:06:04.087480 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:04.087440 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f0ef4-s9tsc" event={"ID":"f7a10320-b197-4471-ac63-fd5041d2a8d4","Type":"ContainerDied","Data":"344e8353e6748bc1d43ea4c3554307d6c38d222ea96c756027bfb940c6b432b1"} Oct 29 23:06:04.328091 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:04.328065 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw" Oct 29 23:06:04.418591 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:04.418557 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0f8724b-cf95-4a2d-8061-cb9236f16c8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f8724b-cf95-4a2d-8061-cb9236f16c8f-maas-api-xks-ca\") pod \"f0f8724b-cf95-4a2d-8061-cb9236f16c8f\" (UID: \"f0f8724b-cf95-4a2d-8061-cb9236f16c8f\") " Oct 29 23:06:04.419585 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:04.419559 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f8724b-cf95-4a2d-8061-cb9236f16c8f-kube-api-access-cmlzp\" (UniqueName: \"kubernetes.io/projected/f0f8724b-cf95-4a2d-8061-cb9236f16c8f-kube-api-access-cmlzp\") pod \"f0f8724b-cf95-4a2d-8061-cb9236f16c8f\" (UID: \"f0f8724b-cf95-4a2d-8061-cb9236f16c8f\") " Oct 29 23:06:04.420261 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:04.420188 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f8724b-cf95-4a2d-8061-cb9236f16c8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f8724b-cf95-4a2d-8061-cb9236f16c8f-maas-api-openshift-ca\") pod \"f0f8724b-cf95-4a2d-8061-cb9236f16c8f\" (UID: \"f0f8724b-cf95-4a2d-8061-cb9236f16c8f\") " Oct 29 23:06:04.420666 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:04.420638 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f8724b-cf95-4a2d-8061-cb9236f16c8f-maas-api-openshift-ca" pod "f0f8724b-cf95-4a2d-8061-cb9236f16c8f" (UID: "f0f8724b-cf95-4a2d-8061-cb9236f16c8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:04.421325 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:04.421299 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f8724b-cf95-4a2d-8061-cb9236f16c8f-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:04.423605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:04.423581 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f8724b-cf95-4a2d-8061-cb9236f16c8f-kube-api-access-cmlzp" pod "f0f8724b-cf95-4a2d-8061-cb9236f16c8f" (UID: "f0f8724b-cf95-4a2d-8061-cb9236f16c8f"). InnerVolumeSpecName "kube-api-access-cmlzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:04.432389 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:04.430504 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0f8724b-cf95-4a2d-8061-cb9236f16c8f-maas-api-xks-ca" pod "f0f8724b-cf95-4a2d-8061-cb9236f16c8f" (UID: "f0f8724b-cf95-4a2d-8061-cb9236f16c8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:04.523091 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:04.523059 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmlzp\" (UniqueName: \"kubernetes.io/projected/f0f8724b-cf95-4a2d-8061-cb9236f16c8f-kube-api-access-cmlzp\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:04.523091 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:04.523091 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f8724b-cf95-4a2d-8061-cb9236f16c8f-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:05.097439 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:05.097405 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw" Oct 29 23:06:05.097932 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:05.097439 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw" event={"ID":"f0f8724b-cf95-4a2d-8061-cb9236f16c8f","Type":"ContainerDied","Data":"93dc0b58fbd9f63f8ea13b12bbd92b84135821a186aff194e244cc4ba6b3803b"} Oct 29 23:06:05.097932 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:05.097513 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93dc0b58fbd9f63f8ea13b12bbd92b84135821a186aff194e244cc4ba6b3803b" Oct 29 23:06:05.256891 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:05.256866 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f0ef4-s9tsc" Oct 29 23:06:05.333318 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:05.333285 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7a10320-b197-4471-ac63-fd5041d2a8d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7a10320-b197-4471-ac63-fd5041d2a8d4-maas-api-openshift-ca\") pod \"f7a10320-b197-4471-ac63-fd5041d2a8d4\" (UID: \"f7a10320-b197-4471-ac63-fd5041d2a8d4\") " Oct 29 23:06:05.333533 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:05.333354 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7a10320-b197-4471-ac63-fd5041d2a8d4-kube-api-access-49gqq\" (UniqueName: \"kubernetes.io/projected/f7a10320-b197-4471-ac63-fd5041d2a8d4-kube-api-access-49gqq\") pod \"f7a10320-b197-4471-ac63-fd5041d2a8d4\" (UID: \"f7a10320-b197-4471-ac63-fd5041d2a8d4\") " Oct 29 23:06:05.333619 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:05.333568 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7a10320-b197-4471-ac63-fd5041d2a8d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7a10320-b197-4471-ac63-fd5041d2a8d4-maas-api-xks-ca\") pod \"f7a10320-b197-4471-ac63-fd5041d2a8d4\" (UID: \"f7a10320-b197-4471-ac63-fd5041d2a8d4\") " Oct 29 23:06:05.333826 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:05.333797 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7a10320-b197-4471-ac63-fd5041d2a8d4-maas-api-openshift-ca" pod "f7a10320-b197-4471-ac63-fd5041d2a8d4" (UID: "f7a10320-b197-4471-ac63-fd5041d2a8d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:05.334542 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:05.334518 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7a10320-b197-4471-ac63-fd5041d2a8d4-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:05.335882 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:05.335847 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a10320-b197-4471-ac63-fd5041d2a8d4-kube-api-access-49gqq" pod "f7a10320-b197-4471-ac63-fd5041d2a8d4" (UID: "f7a10320-b197-4471-ac63-fd5041d2a8d4"). InnerVolumeSpecName "kube-api-access-49gqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:05.336283 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:05.336260 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7a10320-b197-4471-ac63-fd5041d2a8d4-maas-api-xks-ca" pod "f7a10320-b197-4471-ac63-fd5041d2a8d4" (UID: "f7a10320-b197-4471-ac63-fd5041d2a8d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:05.436587 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:05.436555 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7a10320-b197-4471-ac63-fd5041d2a8d4-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:05.436587 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:05.436584 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49gqq\" (UniqueName: \"kubernetes.io/projected/f7a10320-b197-4471-ac63-fd5041d2a8d4-kube-api-access-49gqq\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:06.107561 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:06.107530 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f0ef4-s9tsc" event={"ID":"f7a10320-b197-4471-ac63-fd5041d2a8d4","Type":"ContainerDied","Data":"587606cd021ff4ad4661a05514c6045d5d28aaf8dfdf36220e08d15b6024b37c"} Oct 29 23:06:06.107923 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:06.107573 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa7f0ef4-s9tsc" Oct 29 23:06:06.107923 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:06.107578 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="587606cd021ff4ad4661a05514c6045d5d28aaf8dfdf36220e08d15b6024b37c" Oct 29 23:06:11.293121 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.293083 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-850a6259-nkqdq"] Oct 29 23:06:11.294042 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.294021 4821 cpu_manager.go:414] "containerMap: removing container" podUID="f7a10320-b197-4471-ac63-fd5041d2a8d4" containerName="revoke-keys" Oct 29 23:06:11.294101 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.294048 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a10320-b197-4471-ac63-fd5041d2a8d4" containerName="revoke-keys" Oct 29 23:06:11.294138 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.294120 4821 cpu_manager.go:414] "containerMap: removing container" podUID="f0f8724b-cf95-4a2d-8061-cb9236f16c8f" containerName="revoke-keys" Oct 29 23:06:11.294138 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.294128 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f8724b-cf95-4a2d-8061-cb9236f16c8f" containerName="revoke-keys" Oct 29 23:06:11.294307 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.294295 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f8724b-cf95-4a2d-8061-cb9236f16c8f" containerName="revoke-keys" Oct 29 23:06:11.294355 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.294318 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7a10320-b197-4471-ac63-fd5041d2a8d4" containerName="revoke-keys" Oct 29 23:06:11.299689 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.299666 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-850a6259-nkqdq" Oct 29 23:06:11.330329 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.327341 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27"] Oct 29 23:06:11.346626 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.346594 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:06:11.347069 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.347038 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:06:11.373788 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.373749 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-850a6259-nkqdq"] Oct 29 23:06:11.373979 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.373831 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn"] Oct 29 23:06:11.378094 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.378056 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1e06ef-fc16-4678-a434-b28506de2eb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-850a6259-nkqdq\" (UID: \"8c1e06ef-fc16-4678-a434-b28506de2eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-850a6259-nkqdq" Oct 29 23:06:11.378291 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.378270 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1e06ef-fc16-4678-a434-b28506de2eb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-850a6259-nkqdq\" (UID: \"8c1e06ef-fc16-4678-a434-b28506de2eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-850a6259-nkqdq" Oct 29 23:06:11.378527 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.378509 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxjcm\" (UniqueName: \"kubernetes.io/projected/8c1e06ef-fc16-4678-a434-b28506de2eb3-kube-api-access-pxjcm\") pod \"maas-api-revoke-sub-850a6259-nkqdq\" (UID: \"8c1e06ef-fc16-4678-a434-b28506de2eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-850a6259-nkqdq" Oct 29 23:06:11.380556 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.380539 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.385130 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.385103 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn"] Oct 29 23:06:11.424095 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.421490 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-047892-openshift-default-dockercfg-ftfxh\"" Oct 29 23:06:11.495259 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.493521 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-podinfo\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.495259 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.493578 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2dz6\" (UniqueName: \"kubernetes.io/projected/ff6b1260-3dbc-4391-a1a9-7c2de989026c-kube-api-access-t2dz6\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.495259 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.493608 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istiod-ca-cert\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.495259 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.493724 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1e06ef-fc16-4678-a434-b28506de2eb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-850a6259-nkqdq\" (UID: \"8c1e06ef-fc16-4678-a434-b28506de2eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-850a6259-nkqdq" Oct 29 23:06:11.495259 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.493752 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1e06ef-fc16-4678-a434-b28506de2eb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-850a6259-nkqdq\" (UID: \"8c1e06ef-fc16-4678-a434-b28506de2eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-850a6259-nkqdq" Oct 29 23:06:11.495259 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.493793 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-data\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.495259 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.493837 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-credential-socket\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.495259 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.493874 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-workload-certs\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.495259 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.493972 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-workload-socket\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.495259 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.494060 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-token\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.495259 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.494137 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-envoy\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.495259 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.494165 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxjcm\" (UniqueName: \"kubernetes.io/projected/8c1e06ef-fc16-4678-a434-b28506de2eb3-kube-api-access-pxjcm\") pod \"maas-api-revoke-sub-850a6259-nkqdq\" (UID: \"8c1e06ef-fc16-4678-a434-b28506de2eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-850a6259-nkqdq" Oct 29 23:06:11.496029 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.495935 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1e06ef-fc16-4678-a434-b28506de2eb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-850a6259-nkqdq\" (UID: \"8c1e06ef-fc16-4678-a434-b28506de2eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-850a6259-nkqdq" Oct 29 23:06:11.503032 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.503002 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1e06ef-fc16-4678-a434-b28506de2eb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-850a6259-nkqdq\" (UID: \"8c1e06ef-fc16-4678-a434-b28506de2eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-850a6259-nkqdq" Oct 29 23:06:11.519977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.519939 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxjcm\" (UniqueName: \"kubernetes.io/projected/8c1e06ef-fc16-4678-a434-b28506de2eb3-kube-api-access-pxjcm\") pod \"maas-api-revoke-sub-850a6259-nkqdq\" (UID: \"8c1e06ef-fc16-4678-a434-b28506de2eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-850a6259-nkqdq" Oct 29 23:06:11.597896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.596062 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-workload-socket\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.597896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.596125 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-token\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.597896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.596227 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-envoy\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.597896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.596398 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-podinfo\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.597896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.596441 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2dz6\" (UniqueName: \"kubernetes.io/projected/ff6b1260-3dbc-4391-a1a9-7c2de989026c-kube-api-access-t2dz6\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.597896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.596470 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istiod-ca-cert\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.597896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.596531 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-data\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.597896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.596578 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-credential-socket\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.597896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.596614 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-workload-certs\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.597896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.597081 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-workload-certs\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.597896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.597360 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-workload-socket\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.598651 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.598628 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istiod-ca-cert\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.600507 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.600485 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-data\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.601333 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.601303 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-credential-socket\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.615010 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.614968 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-podinfo\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.622286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.621802 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-envoy\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.626795 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.626763 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-token\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.630306 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.630277 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2dz6\" (UniqueName: \"kubernetes.io/projected/ff6b1260-3dbc-4391-a1a9-7c2de989026c-kube-api-access-t2dz6\") pod \"e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:11.661275 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.661241 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-850a6259-nkqdq" Oct 29 23:06:11.716967 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:11.716725 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:12.103993 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:12.103947 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-850a6259-nkqdq"] Oct 29 23:06:12.126875 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:12.126840 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn"] Oct 29 23:06:12.135557 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:06:12.135532 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6b1260_3dbc_4391_a1a9_7c2de989026c.slice/crio-cb3d2d773e2cf3488bdc65297c23aa746085e3e662400b5457c9021333d4cac2 WatchSource:0}: Error finding container cb3d2d773e2cf3488bdc65297c23aa746085e3e662400b5457c9021333d4cac2: Status 404 returned error can't find the container with id cb3d2d773e2cf3488bdc65297c23aa746085e3e662400b5457c9021333d4cac2 Oct 29 23:06:12.153472 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:12.150698 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:06:12.153472 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:12.150788 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:06:12.153472 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:12.150834 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:06:12.176171 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:12.176133 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" event={"ID":"ff6b1260-3dbc-4391-a1a9-7c2de989026c","Type":"ContainerStarted","Data":"cb3d2d773e2cf3488bdc65297c23aa746085e3e662400b5457c9021333d4cac2"} Oct 29 23:06:12.176379 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:12.176194 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-850a6259-nkqdq" event={"ID":"8c1e06ef-fc16-4678-a434-b28506de2eb3","Type":"ContainerStarted","Data":"18df08a06561c6ec778245c9580da15ad0793da830cb91f689a507fc4c945f1d"} Oct 29 23:06:12.692933 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:12.692842 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc1eb15-s8ttb"] Oct 29 23:06:12.697114 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:12.697096 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc1eb15-s8ttb" Oct 29 23:06:12.720652 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:12.720620 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc1eb15-s8ttb"] Oct 29 23:06:12.777698 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:12.777662 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d33a5db-ede9-4b1c-bc8d-d436e87add8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dc1eb15-s8ttb\" (UID: \"7d33a5db-ede9-4b1c-bc8d-d436e87add8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc1eb15-s8ttb" Oct 29 23:06:12.777843 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:12.777719 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d33a5db-ede9-4b1c-bc8d-d436e87add8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dc1eb15-s8ttb\" (UID: \"7d33a5db-ede9-4b1c-bc8d-d436e87add8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc1eb15-s8ttb" Oct 29 23:06:12.777843 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:12.777807 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-459kb\" (UniqueName: \"kubernetes.io/projected/7d33a5db-ede9-4b1c-bc8d-d436e87add8a-kube-api-access-459kb\") pod \"maas-api-revoke-sub-9dc1eb15-s8ttb\" (UID: \"7d33a5db-ede9-4b1c-bc8d-d436e87add8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc1eb15-s8ttb" Oct 29 23:06:12.879706 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:12.879670 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d33a5db-ede9-4b1c-bc8d-d436e87add8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dc1eb15-s8ttb\" (UID: \"7d33a5db-ede9-4b1c-bc8d-d436e87add8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc1eb15-s8ttb" Oct 29 23:06:12.879706 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:12.879719 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d33a5db-ede9-4b1c-bc8d-d436e87add8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dc1eb15-s8ttb\" (UID: \"7d33a5db-ede9-4b1c-bc8d-d436e87add8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc1eb15-s8ttb" Oct 29 23:06:12.879981 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:12.879790 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-459kb\" (UniqueName: \"kubernetes.io/projected/7d33a5db-ede9-4b1c-bc8d-d436e87add8a-kube-api-access-459kb\") pod \"maas-api-revoke-sub-9dc1eb15-s8ttb\" (UID: \"7d33a5db-ede9-4b1c-bc8d-d436e87add8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc1eb15-s8ttb" Oct 29 23:06:12.880497 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:12.880468 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d33a5db-ede9-4b1c-bc8d-d436e87add8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dc1eb15-s8ttb\" (UID: \"7d33a5db-ede9-4b1c-bc8d-d436e87add8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc1eb15-s8ttb" Oct 29 23:06:12.882399 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:12.882369 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d33a5db-ede9-4b1c-bc8d-d436e87add8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dc1eb15-s8ttb\" (UID: \"7d33a5db-ede9-4b1c-bc8d-d436e87add8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc1eb15-s8ttb" Oct 29 23:06:12.904229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:12.904191 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-459kb\" (UniqueName: \"kubernetes.io/projected/7d33a5db-ede9-4b1c-bc8d-d436e87add8a-kube-api-access-459kb\") pod \"maas-api-revoke-sub-9dc1eb15-s8ttb\" (UID: \"7d33a5db-ede9-4b1c-bc8d-d436e87add8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc1eb15-s8ttb" Oct 29 23:06:13.013751 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:13.013666 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc1eb15-s8ttb" Oct 29 23:06:13.190950 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:13.190869 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-850a6259-nkqdq" event={"ID":"8c1e06ef-fc16-4678-a434-b28506de2eb3","Type":"ContainerStarted","Data":"6913c8e965dfc17346b6f783985fae0facbc5e53e6f590a235c1185b62ae24a8"} Oct 29 23:06:13.194035 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:13.193997 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" event={"ID":"ff6b1260-3dbc-4391-a1a9-7c2de989026c","Type":"ContainerStarted","Data":"59b5413d902abc9442d1e9976f6e4c306a237d1d9b34c974b04adf31e18bbb64"} Oct 29 23:06:13.235977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:13.235926 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-850a6259-nkqdq" podStartSLOduration=2.235911562 podStartE2EDuration="2.235911562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:13.232687545 +0000 UTC m=+1427.173235455" watchObservedRunningTime="2026-10-29 23:06:13.235911562 +0000 UTC m=+1427.176459503" Oct 29 23:06:13.283974 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:13.283870 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" podStartSLOduration=2.283854164 podStartE2EDuration="2.283854164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:13.279591965 +0000 UTC m=+1427.220139876" watchObservedRunningTime="2026-10-29 23:06:13.283854164 +0000 UTC m=+1427.224402072" Oct 29 23:06:13.343908 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:13.343876 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc1eb15-s8ttb"] Oct 29 23:06:13.344059 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:06:13.343983 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d33a5db_ede9_4b1c_bc8d_d436e87add8a.slice/crio-7b9f5a83ca8c9639fc267d44157dbf09e70a6301dd00052e41ca5c33a6ea846c WatchSource:0}: Error finding container 7b9f5a83ca8c9639fc267d44157dbf09e70a6301dd00052e41ca5c33a6ea846c: Status 404 returned error can't find the container with id 7b9f5a83ca8c9639fc267d44157dbf09e70a6301dd00052e41ca5c33a6ea846c Oct 29 23:06:13.728681 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:13.728653 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:13.735263 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:13.735237 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:14.203582 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:14.203550 4821 generic.go:356] "Generic (PLEG): container finished" podID="8c1e06ef-fc16-4678-a434-b28506de2eb3" containerID="6913c8e965dfc17346b6f783985fae0facbc5e53e6f590a235c1185b62ae24a8" exitCode=0 Oct 29 23:06:14.203759 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:14.203639 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-850a6259-nkqdq" event={"ID":"8c1e06ef-fc16-4678-a434-b28506de2eb3","Type":"ContainerDied","Data":"6913c8e965dfc17346b6f783985fae0facbc5e53e6f590a235c1185b62ae24a8"} Oct 29 23:06:14.205479 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:14.205451 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc1eb15-s8ttb" event={"ID":"7d33a5db-ede9-4b1c-bc8d-d436e87add8a","Type":"ContainerStarted","Data":"cb4ff358af0460faa4bc34c4180bfba400997e621580404b95a31eaba5b6e002"} Oct 29 23:06:14.205596 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:14.205494 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc1eb15-s8ttb" event={"ID":"7d33a5db-ede9-4b1c-bc8d-d436e87add8a","Type":"ContainerStarted","Data":"7b9f5a83ca8c9639fc267d44157dbf09e70a6301dd00052e41ca5c33a6ea846c"} Oct 29 23:06:14.205782 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:14.205768 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:14.206591 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:14.206573 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:06:14.289451 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:14.289401 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc1eb15-s8ttb" podStartSLOduration=2.289385267 podStartE2EDuration="2.289385267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:14.288018645 +0000 UTC m=+1428.228566556" watchObservedRunningTime="2026-10-29 23:06:14.289385267 +0000 UTC m=+1428.229933177" Oct 29 23:06:15.215953 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.215916 4821 generic.go:356] "Generic (PLEG): container finished" podID="7d33a5db-ede9-4b1c-bc8d-d436e87add8a" containerID="cb4ff358af0460faa4bc34c4180bfba400997e621580404b95a31eaba5b6e002" exitCode=0 Oct 29 23:06:15.216427 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.216001 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc1eb15-s8ttb" event={"ID":"7d33a5db-ede9-4b1c-bc8d-d436e87add8a","Type":"ContainerDied","Data":"cb4ff358af0460faa4bc34c4180bfba400997e621580404b95a31eaba5b6e002"} Oct 29 23:06:15.403110 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.403084 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-850a6259-nkqdq" Oct 29 23:06:15.521865 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.521778 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c1e06ef-fc16-4678-a434-b28506de2eb3-kube-api-access-pxjcm\" (UniqueName: \"kubernetes.io/projected/8c1e06ef-fc16-4678-a434-b28506de2eb3-kube-api-access-pxjcm\") pod \"8c1e06ef-fc16-4678-a434-b28506de2eb3\" (UID: \"8c1e06ef-fc16-4678-a434-b28506de2eb3\") " Oct 29 23:06:15.521865 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.521811 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c1e06ef-fc16-4678-a434-b28506de2eb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1e06ef-fc16-4678-a434-b28506de2eb3-maas-api-xks-ca\") pod \"8c1e06ef-fc16-4678-a434-b28506de2eb3\" (UID: \"8c1e06ef-fc16-4678-a434-b28506de2eb3\") " Oct 29 23:06:15.522031 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.521922 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c1e06ef-fc16-4678-a434-b28506de2eb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1e06ef-fc16-4678-a434-b28506de2eb3-maas-api-openshift-ca\") pod \"8c1e06ef-fc16-4678-a434-b28506de2eb3\" (UID: \"8c1e06ef-fc16-4678-a434-b28506de2eb3\") " Oct 29 23:06:15.522340 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.522315 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c1e06ef-fc16-4678-a434-b28506de2eb3-maas-api-openshift-ca" pod "8c1e06ef-fc16-4678-a434-b28506de2eb3" (UID: "8c1e06ef-fc16-4678-a434-b28506de2eb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:15.522562 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.522546 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1e06ef-fc16-4678-a434-b28506de2eb3-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:15.524616 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.524585 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c1e06ef-fc16-4678-a434-b28506de2eb3-maas-api-xks-ca" pod "8c1e06ef-fc16-4678-a434-b28506de2eb3" (UID: "8c1e06ef-fc16-4678-a434-b28506de2eb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:15.524616 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.524601 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c1e06ef-fc16-4678-a434-b28506de2eb3-kube-api-access-pxjcm" pod "8c1e06ef-fc16-4678-a434-b28506de2eb3" (UID: "8c1e06ef-fc16-4678-a434-b28506de2eb3"). InnerVolumeSpecName "kube-api-access-pxjcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:15.624401 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.624369 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxjcm\" (UniqueName: \"kubernetes.io/projected/8c1e06ef-fc16-4678-a434-b28506de2eb3-kube-api-access-pxjcm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:15.624401 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.624396 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1e06ef-fc16-4678-a434-b28506de2eb3-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:15.721051 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.721016 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g"] Oct 29 23:06:15.721673 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.721658 4821 cpu_manager.go:414] "containerMap: removing container" podUID="8c1e06ef-fc16-4678-a434-b28506de2eb3" containerName="revoke-keys" Oct 29 23:06:15.721726 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.721675 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c1e06ef-fc16-4678-a434-b28506de2eb3" containerName="revoke-keys" Oct 29 23:06:15.721798 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.721789 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c1e06ef-fc16-4678-a434-b28506de2eb3" containerName="revoke-keys" Oct 29 23:06:15.725537 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.725438 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g" Oct 29 23:06:15.744860 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.744618 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-77238f-serving-cert\"" Oct 29 23:06:15.747976 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.747954 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g"] Oct 29 23:06:15.827827 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.827734 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/562cc122-2430-4052-9085-f67e43a9cbb9-metrics-certs\") pod \"maas-api-e2e-ipp-b-77238f-85645c556-kxs5g\" (UID: \"562cc122-2430-4052-9085-f67e43a9cbb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g" Oct 29 23:06:15.827827 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.827793 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl5rh\" (UniqueName: \"kubernetes.io/projected/562cc122-2430-4052-9085-f67e43a9cbb9-kube-api-access-pl5rh\") pod \"maas-api-e2e-ipp-b-77238f-85645c556-kxs5g\" (UID: \"562cc122-2430-4052-9085-f67e43a9cbb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g" Oct 29 23:06:15.828039 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.827938 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/562cc122-2430-4052-9085-f67e43a9cbb9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-77238f-85645c556-kxs5g\" (UID: \"562cc122-2430-4052-9085-f67e43a9cbb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g" Oct 29 23:06:15.828039 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.827974 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/562cc122-2430-4052-9085-f67e43a9cbb9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-77238f-85645c556-kxs5g\" (UID: \"562cc122-2430-4052-9085-f67e43a9cbb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g" Oct 29 23:06:15.929901 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.929852 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/562cc122-2430-4052-9085-f67e43a9cbb9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-77238f-85645c556-kxs5g\" (UID: \"562cc122-2430-4052-9085-f67e43a9cbb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g" Oct 29 23:06:15.930111 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.929916 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/562cc122-2430-4052-9085-f67e43a9cbb9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-77238f-85645c556-kxs5g\" (UID: \"562cc122-2430-4052-9085-f67e43a9cbb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g" Oct 29 23:06:15.930311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.930182 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/562cc122-2430-4052-9085-f67e43a9cbb9-metrics-certs\") pod \"maas-api-e2e-ipp-b-77238f-85645c556-kxs5g\" (UID: \"562cc122-2430-4052-9085-f67e43a9cbb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g" Oct 29 23:06:15.930311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.930264 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl5rh\" (UniqueName: \"kubernetes.io/projected/562cc122-2430-4052-9085-f67e43a9cbb9-kube-api-access-pl5rh\") pod \"maas-api-e2e-ipp-b-77238f-85645c556-kxs5g\" (UID: \"562cc122-2430-4052-9085-f67e43a9cbb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g" Oct 29 23:06:15.930781 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.930753 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/562cc122-2430-4052-9085-f67e43a9cbb9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-77238f-85645c556-kxs5g\" (UID: \"562cc122-2430-4052-9085-f67e43a9cbb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g" Oct 29 23:06:15.933402 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.933149 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/562cc122-2430-4052-9085-f67e43a9cbb9-metrics-certs\") pod \"maas-api-e2e-ipp-b-77238f-85645c556-kxs5g\" (UID: \"562cc122-2430-4052-9085-f67e43a9cbb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g" Oct 29 23:06:15.933402 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.933178 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/562cc122-2430-4052-9085-f67e43a9cbb9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-77238f-85645c556-kxs5g\" (UID: \"562cc122-2430-4052-9085-f67e43a9cbb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g" Oct 29 23:06:15.955403 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:15.955382 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl5rh\" (UniqueName: \"kubernetes.io/projected/562cc122-2430-4052-9085-f67e43a9cbb9-kube-api-access-pl5rh\") pod \"maas-api-e2e-ipp-b-77238f-85645c556-kxs5g\" (UID: \"562cc122-2430-4052-9085-f67e43a9cbb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g" Oct 29 23:06:16.040277 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:16.040236 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g" Oct 29 23:06:16.310363 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:16.309348 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-850a6259-nkqdq" Oct 29 23:06:16.310363 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:16.309814 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-850a6259-nkqdq" event={"ID":"8c1e06ef-fc16-4678-a434-b28506de2eb3","Type":"ContainerDied","Data":"18df08a06561c6ec778245c9580da15ad0793da830cb91f689a507fc4c945f1d"} Oct 29 23:06:16.310363 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:16.309869 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18df08a06561c6ec778245c9580da15ad0793da830cb91f689a507fc4c945f1d" Oct 29 23:06:16.328846 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:16.328811 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27" podUID="0ae12788-66ec-4e5f-9248-00df150bba93" containerName="payload-processing" containerID="cri-o://4db4e9d4c95111ef3c390ae80c3373d03529f8319ded017eb00779aeae21508a" gracePeriod=25 Oct 29 23:06:16.499334 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:16.499301 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g"] Oct 29 23:06:16.538609 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:16.538586 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc1eb15-s8ttb" Oct 29 23:06:16.642490 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:16.642466 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27" Oct 29 23:06:16.706171 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:16.706143 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d33a5db-ede9-4b1c-bc8d-d436e87add8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d33a5db-ede9-4b1c-bc8d-d436e87add8a-maas-api-openshift-ca\") pod \"7d33a5db-ede9-4b1c-bc8d-d436e87add8a\" (UID: \"7d33a5db-ede9-4b1c-bc8d-d436e87add8a\") " Oct 29 23:06:16.706360 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:16.706349 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d33a5db-ede9-4b1c-bc8d-d436e87add8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d33a5db-ede9-4b1c-bc8d-d436e87add8a-maas-api-xks-ca\") pod \"7d33a5db-ede9-4b1c-bc8d-d436e87add8a\" (UID: \"7d33a5db-ede9-4b1c-bc8d-d436e87add8a\") " Oct 29 23:06:16.706457 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:16.706416 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d33a5db-ede9-4b1c-bc8d-d436e87add8a-kube-api-access-459kb\" (UniqueName: \"kubernetes.io/projected/7d33a5db-ede9-4b1c-bc8d-d436e87add8a-kube-api-access-459kb\") pod \"7d33a5db-ede9-4b1c-bc8d-d436e87add8a\" (UID: \"7d33a5db-ede9-4b1c-bc8d-d436e87add8a\") " Oct 29 23:06:16.706563 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:16.706536 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d33a5db-ede9-4b1c-bc8d-d436e87add8a-maas-api-openshift-ca" pod "7d33a5db-ede9-4b1c-bc8d-d436e87add8a" (UID: "7d33a5db-ede9-4b1c-bc8d-d436e87add8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:16.707113 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:16.707057 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d33a5db-ede9-4b1c-bc8d-d436e87add8a-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:16.709134 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:16.709112 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d33a5db-ede9-4b1c-bc8d-d436e87add8a-maas-api-xks-ca" pod "7d33a5db-ede9-4b1c-bc8d-d436e87add8a" (UID: "7d33a5db-ede9-4b1c-bc8d-d436e87add8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:16.709274 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:16.709255 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d33a5db-ede9-4b1c-bc8d-d436e87add8a-kube-api-access-459kb" pod "7d33a5db-ede9-4b1c-bc8d-d436e87add8a" (UID: "7d33a5db-ede9-4b1c-bc8d-d436e87add8a"). InnerVolumeSpecName "kube-api-access-459kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:16.808260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:16.808148 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ae12788-66ec-4e5f-9248-00df150bba93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae12788-66ec-4e5f-9248-00df150bba93-plugins-config-volume\") pod \"0ae12788-66ec-4e5f-9248-00df150bba93\" (UID: \"0ae12788-66ec-4e5f-9248-00df150bba93\") " Oct 29 23:06:16.808414 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:16.808340 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ae12788-66ec-4e5f-9248-00df150bba93-kube-api-access-4h4hj\" (UniqueName: \"kubernetes.io/projected/0ae12788-66ec-4e5f-9248-00df150bba93-kube-api-access-4h4hj\") pod \"0ae12788-66ec-4e5f-9248-00df150bba93\" (UID: \"0ae12788-66ec-4e5f-9248-00df150bba93\") " Oct 29 23:06:16.808518 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:16.808494 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ae12788-66ec-4e5f-9248-00df150bba93-plugins-config-volume" pod "0ae12788-66ec-4e5f-9248-00df150bba93" (UID: "0ae12788-66ec-4e5f-9248-00df150bba93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:16.808862 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:16.808846 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae12788-66ec-4e5f-9248-00df150bba93-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:16.808928 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:16.808865 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d33a5db-ede9-4b1c-bc8d-d436e87add8a-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:16.808928 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:16.808874 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-459kb\" (UniqueName: \"kubernetes.io/projected/7d33a5db-ede9-4b1c-bc8d-d436e87add8a-kube-api-access-459kb\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:16.810739 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:16.810716 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae12788-66ec-4e5f-9248-00df150bba93-kube-api-access-4h4hj" pod "0ae12788-66ec-4e5f-9248-00df150bba93" (UID: "0ae12788-66ec-4e5f-9248-00df150bba93"). InnerVolumeSpecName "kube-api-access-4h4hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:16.910801 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:16.910768 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h4hj\" (UniqueName: \"kubernetes.io/projected/0ae12788-66ec-4e5f-9248-00df150bba93-kube-api-access-4h4hj\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:17.320920 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.320882 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc1eb15-s8ttb" Oct 29 23:06:17.320920 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.320901 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dc1eb15-s8ttb" event={"ID":"7d33a5db-ede9-4b1c-bc8d-d436e87add8a","Type":"ContainerDied","Data":"7b9f5a83ca8c9639fc267d44157dbf09e70a6301dd00052e41ca5c33a6ea846c"} Oct 29 23:06:17.321445 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.320958 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b9f5a83ca8c9639fc267d44157dbf09e70a6301dd00052e41ca5c33a6ea846c" Oct 29 23:06:17.322956 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.322925 4821 generic.go:356] "Generic (PLEG): container finished" podID="0ae12788-66ec-4e5f-9248-00df150bba93" containerID="4db4e9d4c95111ef3c390ae80c3373d03529f8319ded017eb00779aeae21508a" exitCode=0 Oct 29 23:06:17.323112 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.322964 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27" Oct 29 23:06:17.323112 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.323015 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27" event={"ID":"0ae12788-66ec-4e5f-9248-00df150bba93","Type":"ContainerDied","Data":"4db4e9d4c95111ef3c390ae80c3373d03529f8319ded017eb00779aeae21508a"} Oct 29 23:06:17.323112 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.323081 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27" event={"ID":"0ae12788-66ec-4e5f-9248-00df150bba93","Type":"ContainerDied","Data":"b56c0ffafe7d0b1af99cc916775ea4bfa8ba05ba1ab5de8252b1684023838ade"} Oct 29 23:06:17.323503 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.323103 4821 scope.go:122] "RemoveContainer" containerID="4db4e9d4c95111ef3c390ae80c3373d03529f8319ded017eb00779aeae21508a" Oct 29 23:06:17.326539 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.326482 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g" event={"ID":"562cc122-2430-4052-9085-f67e43a9cbb9","Type":"ContainerStarted","Data":"d8b7ed801a9c00ac2a55933540578c0efccd42bf88b9d08ea323ffe8412c421a"} Oct 29 23:06:17.326539 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.326528 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g" event={"ID":"562cc122-2430-4052-9085-f67e43a9cbb9","Type":"ContainerStarted","Data":"33201b517ca8c47eb1d72f59b140ef2c291b40515583f6120602742b0e0a4ab3"} Oct 29 23:06:17.326883 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.326868 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g" Oct 29 23:06:17.344306 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.344276 4821 scope.go:122] "RemoveContainer" containerID="4db4e9d4c95111ef3c390ae80c3373d03529f8319ded017eb00779aeae21508a" Oct 29 23:06:17.345399 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:06:17.345332 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4db4e9d4c95111ef3c390ae80c3373d03529f8319ded017eb00779aeae21508a\": container with ID starting with 4db4e9d4c95111ef3c390ae80c3373d03529f8319ded017eb00779aeae21508a not found: ID does not exist" containerID="4db4e9d4c95111ef3c390ae80c3373d03529f8319ded017eb00779aeae21508a" Oct 29 23:06:17.345399 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.345371 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db4e9d4c95111ef3c390ae80c3373d03529f8319ded017eb00779aeae21508a"} err="failed to get container status \"4db4e9d4c95111ef3c390ae80c3373d03529f8319ded017eb00779aeae21508a\": rpc error: code = NotFound desc = could not find container \"4db4e9d4c95111ef3c390ae80c3373d03529f8319ded017eb00779aeae21508a\": container with ID starting with 4db4e9d4c95111ef3c390ae80c3373d03529f8319ded017eb00779aeae21508a not found: ID does not exist" Oct 29 23:06:17.390373 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.390307 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g" podStartSLOduration=1.936164663 podStartE2EDuration="2.390287709s" totalImagesPullingTime="454.123046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:17.387669233 +0000 UTC m=+1431.328217144" watchObservedRunningTime="2026-10-29 23:06:17.390287709 +0000 UTC m=+1431.330835621" Oct 29 23:06:17.438867 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.438840 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27"] Oct 29 23:06:17.445568 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.445543 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-14debc-68b89977d6-thr27"] Oct 29 23:06:17.778834 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.778801 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8"] Oct 29 23:06:17.779424 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.779410 4821 cpu_manager.go:414] "containerMap: removing container" podUID="7d33a5db-ede9-4b1c-bc8d-d436e87add8a" containerName="revoke-keys" Oct 29 23:06:17.779492 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.779425 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d33a5db-ede9-4b1c-bc8d-d436e87add8a" containerName="revoke-keys" Oct 29 23:06:17.779492 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.779450 4821 cpu_manager.go:414] "containerMap: removing container" podUID="0ae12788-66ec-4e5f-9248-00df150bba93" containerName="payload-processing" Oct 29 23:06:17.779492 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.779456 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ae12788-66ec-4e5f-9248-00df150bba93" containerName="payload-processing" Oct 29 23:06:17.779586 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.779562 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ae12788-66ec-4e5f-9248-00df150bba93" containerName="payload-processing" Oct 29 23:06:17.779586 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.779576 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d33a5db-ede9-4b1c-bc8d-d436e87add8a" containerName="revoke-keys" Oct 29 23:06:17.784534 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.784515 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8" Oct 29 23:06:17.811586 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.811561 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-047892\"" Oct 29 23:06:17.811689 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.811604 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-047892-dockercfg-lndw5\"" Oct 29 23:06:17.821375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.821177 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8"] Oct 29 23:06:17.838040 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.837987 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee57d28-4336-4792-aeeb-044ee8338a9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8\" (UID: \"bee57d28-4336-4792-aeeb-044ee8338a9b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8" Oct 29 23:06:17.838274 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.838075 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf8ml\" (UniqueName: \"kubernetes.io/projected/bee57d28-4336-4792-aeeb-044ee8338a9b-kube-api-access-kf8ml\") pod \"payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8\" (UID: \"bee57d28-4336-4792-aeeb-044ee8338a9b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8" Oct 29 23:06:17.990221 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.980815 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee57d28-4336-4792-aeeb-044ee8338a9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8\" (UID: \"bee57d28-4336-4792-aeeb-044ee8338a9b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8" Oct 29 23:06:17.990221 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.980913 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf8ml\" (UniqueName: \"kubernetes.io/projected/bee57d28-4336-4792-aeeb-044ee8338a9b-kube-api-access-kf8ml\") pod \"payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8\" (UID: \"bee57d28-4336-4792-aeeb-044ee8338a9b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8" Oct 29 23:06:17.990221 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.985539 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee57d28-4336-4792-aeeb-044ee8338a9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8\" (UID: \"bee57d28-4336-4792-aeeb-044ee8338a9b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8" Oct 29 23:06:17.990740 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:17.990714 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475"] Oct 29 23:06:18.032725 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:18.019527 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475" Oct 29 23:06:18.032725 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:18.025459 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475"] Oct 29 23:06:18.097231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:18.094516 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca53539d-ef76-4a59-8b01-815fe110a113-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475\" (UID: \"ca53539d-ef76-4a59-8b01-815fe110a113\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475" Oct 29 23:06:18.097231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:18.094795 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckpq7\" (UniqueName: \"kubernetes.io/projected/ca53539d-ef76-4a59-8b01-815fe110a113-kube-api-access-ckpq7\") pod \"payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475\" (UID: \"ca53539d-ef76-4a59-8b01-815fe110a113\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475" Oct 29 23:06:18.145303 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:18.145273 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf8ml\" (UniqueName: \"kubernetes.io/projected/bee57d28-4336-4792-aeeb-044ee8338a9b-kube-api-access-kf8ml\") pod \"payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8\" (UID: \"bee57d28-4336-4792-aeeb-044ee8338a9b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8" Oct 29 23:06:18.182160 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:18.182109 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ae12788-66ec-4e5f-9248-00df150bba93" path="/var/lib/kubelet/pods/0ae12788-66ec-4e5f-9248-00df150bba93/volumes" Oct 29 23:06:18.197231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:18.196575 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckpq7\" (UniqueName: \"kubernetes.io/projected/ca53539d-ef76-4a59-8b01-815fe110a113-kube-api-access-ckpq7\") pod \"payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475\" (UID: \"ca53539d-ef76-4a59-8b01-815fe110a113\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475" Oct 29 23:06:18.197231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:18.196752 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca53539d-ef76-4a59-8b01-815fe110a113-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475\" (UID: \"ca53539d-ef76-4a59-8b01-815fe110a113\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475" Oct 29 23:06:18.203225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:18.198939 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca53539d-ef76-4a59-8b01-815fe110a113-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475\" (UID: \"ca53539d-ef76-4a59-8b01-815fe110a113\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475" Oct 29 23:06:18.231251 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:18.229818 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckpq7\" (UniqueName: \"kubernetes.io/projected/ca53539d-ef76-4a59-8b01-815fe110a113-kube-api-access-ckpq7\") pod \"payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475\" (UID: \"ca53539d-ef76-4a59-8b01-815fe110a113\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475" Oct 29 23:06:18.400628 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:18.400595 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8" Oct 29 23:06:18.467724 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:18.467698 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475" Oct 29 23:06:18.875607 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:18.871666 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8"] Oct 29 23:06:18.876819 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:06:18.876780 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee57d28_4336_4792_aeeb_044ee8338a9b.slice/crio-069f203d93785598f5d006c6938f2c1c473efac2096ce44ca5589d99e894dcba WatchSource:0}: Error finding container 069f203d93785598f5d006c6938f2c1c473efac2096ce44ca5589d99e894dcba: Status 404 returned error can't find the container with id 069f203d93785598f5d006c6938f2c1c473efac2096ce44ca5589d99e894dcba Oct 29 23:06:19.093612 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:19.093570 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw"] Oct 29 23:06:19.107455 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:19.107420 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-14debc-sbpcw"] Oct 29 23:06:19.115510 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:06:19.115479 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca53539d_ef76_4a59_8b01_815fe110a113.slice/crio-982afd22f6b0aa17c493aa1292d57c81d5d838492be2218b2fafdab97b8566d8 WatchSource:0}: Error finding container 982afd22f6b0aa17c493aa1292d57c81d5d838492be2218b2fafdab97b8566d8: Status 404 returned error can't find the container with id 982afd22f6b0aa17c493aa1292d57c81d5d838492be2218b2fafdab97b8566d8 Oct 29 23:06:19.116582 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:19.116180 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475"] Oct 29 23:06:19.359842 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:19.359802 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8" event={"ID":"bee57d28-4336-4792-aeeb-044ee8338a9b","Type":"ContainerStarted","Data":"b7a212ae818443ffc011b3bb557c5741b863678db7d8d5f3e844357d86e8779c"} Oct 29 23:06:19.360048 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:19.359865 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8" event={"ID":"bee57d28-4336-4792-aeeb-044ee8338a9b","Type":"ContainerStarted","Data":"069f203d93785598f5d006c6938f2c1c473efac2096ce44ca5589d99e894dcba"} Oct 29 23:06:19.360113 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:19.360084 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8" Oct 29 23:06:19.362389 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:19.362358 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475" event={"ID":"ca53539d-ef76-4a59-8b01-815fe110a113","Type":"ContainerStarted","Data":"fe248149a0611f21bc014106d80ecc0bb6000bc628ff0057852d079b457dc33e"} Oct 29 23:06:19.362528 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:19.362410 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475" event={"ID":"ca53539d-ef76-4a59-8b01-815fe110a113","Type":"ContainerStarted","Data":"982afd22f6b0aa17c493aa1292d57c81d5d838492be2218b2fafdab97b8566d8"} Oct 29 23:06:19.362596 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:19.362531 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475" Oct 29 23:06:19.423275 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:19.423001 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8" podStartSLOduration=2.422979271 podStartE2EDuration="2.422979271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:19.40891295 +0000 UTC m=+1433.349460871" watchObservedRunningTime="2026-10-29 23:06:19.422979271 +0000 UTC m=+1433.363527186" Oct 29 23:06:19.462925 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:19.462806 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475" podStartSLOduration=2.462787278 podStartE2EDuration="2.462787278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:19.456719499 +0000 UTC m=+1433.397267410" watchObservedRunningTime="2026-10-29 23:06:19.462787278 +0000 UTC m=+1433.403335188" Oct 29 23:06:20.171563 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:20.171522 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0f8724b-cf95-4a2d-8061-cb9236f16c8f" path="/var/lib/kubelet/pods/f0f8724b-cf95-4a2d-8061-cb9236f16c8f/volumes" Oct 29 23:06:23.351080 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:23.351050 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g" Oct 29 23:06:23.888286 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:06:23.888243 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:06:26.351372 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:26.351329 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4"] Oct 29 23:06:26.352017 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:26.351615 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" podUID="d654f3f3-8c6b-46e4-b528-0459173f9d2a" containerName="istio-proxy" containerID="cri-o://02227b5210bf0c78963aa280342b6533c0b18c672d0cba6c3d8935f546982b3f" gracePeriod=30 Oct 29 23:06:26.856391 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:26.856356 4821 scope.go:122] "RemoveContainer" containerID="c43c9e058ffb72c1a0d7396903a814e8e81cb154f3940fb3dd0dd845c0435f5e" Oct 29 23:06:26.868246 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:26.868181 4821 scope.go:122] "RemoveContainer" containerID="e83be1af553a11aa74cd2f214051b2ddf8debf14c1216f25f7c72c5d2a682a19" Oct 29 23:06:26.967887 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:26.967858 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd"] Oct 29 23:06:26.973626 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:26.973607 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd" Oct 29 23:06:26.993721 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:26.993694 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd"] Oct 29 23:06:27.022267 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:27.022226 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/324e8f24-448e-4b50-ba97-20abef961f80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd\" (UID: \"324e8f24-448e-4b50-ba97-20abef961f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd" Oct 29 23:06:27.022445 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:27.022419 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/324e8f24-448e-4b50-ba97-20abef961f80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd\" (UID: \"324e8f24-448e-4b50-ba97-20abef961f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd" Oct 29 23:06:27.022517 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:27.022459 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r24kw\" (UniqueName: \"kubernetes.io/projected/324e8f24-448e-4b50-ba97-20abef961f80-kube-api-access-r24kw\") pod \"maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd\" (UID: \"324e8f24-448e-4b50-ba97-20abef961f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd" Oct 29 23:06:27.029217 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:27.029172 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:06:27.029348 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:27.029244 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:06:27.124193 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:27.124156 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/324e8f24-448e-4b50-ba97-20abef961f80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd\" (UID: \"324e8f24-448e-4b50-ba97-20abef961f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd" Oct 29 23:06:27.124193 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:27.124220 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r24kw\" (UniqueName: \"kubernetes.io/projected/324e8f24-448e-4b50-ba97-20abef961f80-kube-api-access-r24kw\") pod \"maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd\" (UID: \"324e8f24-448e-4b50-ba97-20abef961f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd" Oct 29 23:06:27.124436 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:27.124421 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/324e8f24-448e-4b50-ba97-20abef961f80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd\" (UID: \"324e8f24-448e-4b50-ba97-20abef961f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd" Oct 29 23:06:27.125040 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:27.125023 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/324e8f24-448e-4b50-ba97-20abef961f80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd\" (UID: \"324e8f24-448e-4b50-ba97-20abef961f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd" Oct 29 23:06:27.126685 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:27.126661 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/324e8f24-448e-4b50-ba97-20abef961f80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd\" (UID: \"324e8f24-448e-4b50-ba97-20abef961f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd" Oct 29 23:06:27.148435 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:27.148413 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r24kw\" (UniqueName: \"kubernetes.io/projected/324e8f24-448e-4b50-ba97-20abef961f80-kube-api-access-r24kw\") pod \"maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd\" (UID: \"324e8f24-448e-4b50-ba97-20abef961f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd" Oct 29 23:06:27.287343 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:27.287310 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd" Oct 29 23:06:27.597176 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:27.597145 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd"] Oct 29 23:06:27.597545 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:06:27.597170 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324e8f24_448e_4b50_ba97_20abef961f80.slice/crio-4a13e398858ce8913178187b0670bde117e93546b8f64005bdb9f999a51420af WatchSource:0}: Error finding container 4a13e398858ce8913178187b0670bde117e93546b8f64005bdb9f999a51420af: Status 404 returned error can't find the container with id 4a13e398858ce8913178187b0670bde117e93546b8f64005bdb9f999a51420af Oct 29 23:06:28.446908 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:28.446867 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd" event={"ID":"324e8f24-448e-4b50-ba97-20abef961f80","Type":"ContainerStarted","Data":"26333c942d94b47d21471b45f2098f920d342ad1d0ed9ce41e90dc0896eea511"} Oct 29 23:06:28.447111 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:28.446929 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd" event={"ID":"324e8f24-448e-4b50-ba97-20abef961f80","Type":"ContainerStarted","Data":"4a13e398858ce8913178187b0670bde117e93546b8f64005bdb9f999a51420af"} Oct 29 23:06:28.492658 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:28.492598 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd" podStartSLOduration=2.492585389 podStartE2EDuration="2.492585389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:28.489796279 +0000 UTC m=+1442.430344189" watchObservedRunningTime="2026-10-29 23:06:28.492585389 +0000 UTC m=+1442.433133342" Oct 29 23:06:29.457160 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:29.457127 4821 generic.go:356] "Generic (PLEG): container finished" podID="324e8f24-448e-4b50-ba97-20abef961f80" containerID="26333c942d94b47d21471b45f2098f920d342ad1d0ed9ce41e90dc0896eea511" exitCode=0 Oct 29 23:06:29.457724 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:29.457173 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd" event={"ID":"324e8f24-448e-4b50-ba97-20abef961f80","Type":"ContainerDied","Data":"26333c942d94b47d21471b45f2098f920d342ad1d0ed9ce41e90dc0896eea511"} Oct 29 23:06:30.374307 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:30.374275 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8" Oct 29 23:06:30.374485 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:30.374351 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475" Oct 29 23:06:30.691667 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:30.691640 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd" Oct 29 23:06:30.787218 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:30.787173 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/324e8f24-448e-4b50-ba97-20abef961f80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/324e8f24-448e-4b50-ba97-20abef961f80-maas-api-openshift-ca\") pod \"324e8f24-448e-4b50-ba97-20abef961f80\" (UID: \"324e8f24-448e-4b50-ba97-20abef961f80\") " Oct 29 23:06:30.787397 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:30.787300 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/324e8f24-448e-4b50-ba97-20abef961f80-kube-api-access-r24kw\" (UniqueName: \"kubernetes.io/projected/324e8f24-448e-4b50-ba97-20abef961f80-kube-api-access-r24kw\") pod \"324e8f24-448e-4b50-ba97-20abef961f80\" (UID: \"324e8f24-448e-4b50-ba97-20abef961f80\") " Oct 29 23:06:30.787512 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:30.787490 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/324e8f24-448e-4b50-ba97-20abef961f80-maas-api-openshift-ca" pod "324e8f24-448e-4b50-ba97-20abef961f80" (UID: "324e8f24-448e-4b50-ba97-20abef961f80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:30.787582 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:30.787564 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/324e8f24-448e-4b50-ba97-20abef961f80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/324e8f24-448e-4b50-ba97-20abef961f80-maas-api-xks-ca\") pod \"324e8f24-448e-4b50-ba97-20abef961f80\" (UID: \"324e8f24-448e-4b50-ba97-20abef961f80\") " Oct 29 23:06:30.789431 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:30.789396 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/324e8f24-448e-4b50-ba97-20abef961f80-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:30.791180 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:30.790970 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324e8f24-448e-4b50-ba97-20abef961f80-kube-api-access-r24kw" pod "324e8f24-448e-4b50-ba97-20abef961f80" (UID: "324e8f24-448e-4b50-ba97-20abef961f80"). InnerVolumeSpecName "kube-api-access-r24kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:30.791180 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:30.791027 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/324e8f24-448e-4b50-ba97-20abef961f80-maas-api-xks-ca" pod "324e8f24-448e-4b50-ba97-20abef961f80" (UID: "324e8f24-448e-4b50-ba97-20abef961f80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:30.891156 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:30.891124 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/324e8f24-448e-4b50-ba97-20abef961f80-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:30.891156 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:30.891153 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r24kw\" (UniqueName: \"kubernetes.io/projected/324e8f24-448e-4b50-ba97-20abef961f80-kube-api-access-r24kw\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.477040 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.477012 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd" Oct 29 23:06:31.477187 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.477032 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd" event={"ID":"324e8f24-448e-4b50-ba97-20abef961f80","Type":"ContainerDied","Data":"4a13e398858ce8913178187b0670bde117e93546b8f64005bdb9f999a51420af"} Oct 29 23:06:31.477187 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.477091 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a13e398858ce8913178187b0670bde117e93546b8f64005bdb9f999a51420af" Oct 29 23:06:31.479505 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.479483 4821 generic.go:356] "Generic (PLEG): container finished" podID="d654f3f3-8c6b-46e4-b528-0459173f9d2a" containerID="02227b5210bf0c78963aa280342b6533c0b18c672d0cba6c3d8935f546982b3f" exitCode=0 Oct 29 23:06:31.479632 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.479542 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" event={"ID":"d654f3f3-8c6b-46e4-b528-0459173f9d2a","Type":"ContainerDied","Data":"02227b5210bf0c78963aa280342b6533c0b18c672d0cba6c3d8935f546982b3f"} Oct 29 23:06:31.479632 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.479595 4821 scope.go:122] "RemoveContainer" containerID="77c8d1d1c29d955bf8d7e8e1b2df5a3fad1b59108a158739d1ccadd269372b55" Oct 29 23:06:31.626144 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.626083 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:06:31.702271 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.702237 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-token\" (UniqueName: \"kubernetes.io/projected/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-token\") pod \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " Oct 29 23:06:31.702781 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.702330 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istiod-ca-cert\") pod \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " Oct 29 23:06:31.702781 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.702359 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-credential-socket\") pod \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " Oct 29 23:06:31.702781 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.702428 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-workload-certs\") pod \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " Oct 29 23:06:31.702781 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.702469 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-envoy\") pod \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " Oct 29 23:06:31.702781 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.702559 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-podinfo\") pod \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " Oct 29 23:06:31.702781 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.702643 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d654f3f3-8c6b-46e4-b528-0459173f9d2a-kube-api-access-l4dbb\" (UniqueName: \"kubernetes.io/projected/d654f3f3-8c6b-46e4-b528-0459173f9d2a-kube-api-access-l4dbb\") pod \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " Oct 29 23:06:31.702781 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.702684 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-workload-socket\") pod \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " Oct 29 23:06:31.702781 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.702754 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-workload-certs" pod "d654f3f3-8c6b-46e4-b528-0459173f9d2a" (UID: "d654f3f3-8c6b-46e4-b528-0459173f9d2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:31.703414 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.702829 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-data\") pod \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\" (UID: \"d654f3f3-8c6b-46e4-b528-0459173f9d2a\") " Oct 29 23:06:31.703414 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.702978 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-workload-socket" pod "d654f3f3-8c6b-46e4-b528-0459173f9d2a" (UID: "d654f3f3-8c6b-46e4-b528-0459173f9d2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:31.703414 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.703149 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istiod-ca-cert" pod "d654f3f3-8c6b-46e4-b528-0459173f9d2a" (UID: "d654f3f3-8c6b-46e4-b528-0459173f9d2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:31.703414 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.703274 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-data" pod "d654f3f3-8c6b-46e4-b528-0459173f9d2a" (UID: "d654f3f3-8c6b-46e4-b528-0459173f9d2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:31.703659 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.703480 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-credential-socket" pod "d654f3f3-8c6b-46e4-b528-0459173f9d2a" (UID: "d654f3f3-8c6b-46e4-b528-0459173f9d2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:31.703732 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.703711 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.703790 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.703738 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.703790 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.703754 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.703790 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.703767 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.703790 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.703779 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.706849 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.706823 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-podinfo" pod "d654f3f3-8c6b-46e4-b528-0459173f9d2a" (UID: "d654f3f3-8c6b-46e4-b528-0459173f9d2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:06:31.706983 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.706915 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-envoy" pod "d654f3f3-8c6b-46e4-b528-0459173f9d2a" (UID: "d654f3f3-8c6b-46e4-b528-0459173f9d2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:31.707045 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.706977 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-token" pod "d654f3f3-8c6b-46e4-b528-0459173f9d2a" (UID: "d654f3f3-8c6b-46e4-b528-0459173f9d2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:31.708108 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.708088 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d654f3f3-8c6b-46e4-b528-0459173f9d2a-kube-api-access-l4dbb" pod "d654f3f3-8c6b-46e4-b528-0459173f9d2a" (UID: "d654f3f3-8c6b-46e4-b528-0459173f9d2a"). InnerVolumeSpecName "kube-api-access-l4dbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:31.805571 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.805532 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.805571 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.805561 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.805571 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.805569 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d654f3f3-8c6b-46e4-b528-0459173f9d2a-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.805571 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:31.805579 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4dbb\" (UniqueName: \"kubernetes.io/projected/d654f3f3-8c6b-46e4-b528-0459173f9d2a-kube-api-access-l4dbb\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:32.489267 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:32.489180 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" Oct 29 23:06:32.489267 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:32.489244 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4" event={"ID":"d654f3f3-8c6b-46e4-b528-0459173f9d2a","Type":"ContainerDied","Data":"16b6af43a8fdfb3599076a1c741a980160f6f3fd34f9fea263810ccfb5931416"} Oct 29 23:06:32.489464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:32.489311 4821 scope.go:122] "RemoveContainer" containerID="02227b5210bf0c78963aa280342b6533c0b18c672d0cba6c3d8935f546982b3f" Oct 29 23:06:32.567870 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:32.567829 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4"] Oct 29 23:06:32.580722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:32.580698 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-14debc-openshift-default-dccf7b48f-2g8l4"] Oct 29 23:06:34.171508 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:34.171479 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d654f3f3-8c6b-46e4-b528-0459173f9d2a" path="/var/lib/kubelet/pods/d654f3f3-8c6b-46e4-b528-0459173f9d2a/volumes" Oct 29 23:06:37.434266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:37.434226 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv"] Oct 29 23:06:41.635031 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:41.634994 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn"] Oct 29 23:06:41.646021 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:41.645993 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7d18f7-8gszn"] Oct 29 23:06:42.170935 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:42.170900 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e15fee17-c158-491c-943d-8ae6ebdbc4d7" path="/var/lib/kubelet/pods/e15fee17-c158-491c-943d-8ae6ebdbc4d7/volumes" Oct 29 23:06:42.435435 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:42.435314 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv" podUID="7e4e8184-e3d8-40ba-865d-64f460d13942" containerName="payload-processing" containerID="cri-o://1fca9d6c5075da2e273cc2ee5413fc5a39d2cda980e331b44987cdf1d6785045" gracePeriod=25 Oct 29 23:06:42.584815 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:42.584782 4821 generic.go:356] "Generic (PLEG): container finished" podID="7e4e8184-e3d8-40ba-865d-64f460d13942" containerID="1fca9d6c5075da2e273cc2ee5413fc5a39d2cda980e331b44987cdf1d6785045" exitCode=0 Oct 29 23:06:42.585022 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:42.584863 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv" event={"ID":"7e4e8184-e3d8-40ba-865d-64f460d13942","Type":"ContainerDied","Data":"1fca9d6c5075da2e273cc2ee5413fc5a39d2cda980e331b44987cdf1d6785045"} Oct 29 23:06:42.722297 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:42.722270 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv" Oct 29 23:06:42.772239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:42.772188 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e4e8184-e3d8-40ba-865d-64f460d13942-kube-api-access-bbt2m\" (UniqueName: \"kubernetes.io/projected/7e4e8184-e3d8-40ba-865d-64f460d13942-kube-api-access-bbt2m\") pod \"7e4e8184-e3d8-40ba-865d-64f460d13942\" (UID: \"7e4e8184-e3d8-40ba-865d-64f460d13942\") " Oct 29 23:06:42.772408 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:42.772330 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e4e8184-e3d8-40ba-865d-64f460d13942-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4e8184-e3d8-40ba-865d-64f460d13942-plugins-config-volume\") pod \"7e4e8184-e3d8-40ba-865d-64f460d13942\" (UID: \"7e4e8184-e3d8-40ba-865d-64f460d13942\") " Oct 29 23:06:42.772669 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:42.772646 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e4e8184-e3d8-40ba-865d-64f460d13942-plugins-config-volume" pod "7e4e8184-e3d8-40ba-865d-64f460d13942" (UID: "7e4e8184-e3d8-40ba-865d-64f460d13942"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:42.772880 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:42.772866 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4e8184-e3d8-40ba-865d-64f460d13942-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:42.774424 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:42.774401 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4e8184-e3d8-40ba-865d-64f460d13942-kube-api-access-bbt2m" pod "7e4e8184-e3d8-40ba-865d-64f460d13942" (UID: "7e4e8184-e3d8-40ba-865d-64f460d13942"). InnerVolumeSpecName "kube-api-access-bbt2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:42.874245 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:42.874176 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbt2m\" (UniqueName: \"kubernetes.io/projected/7e4e8184-e3d8-40ba-865d-64f460d13942-kube-api-access-bbt2m\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:43.596071 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:43.596034 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv" event={"ID":"7e4e8184-e3d8-40ba-865d-64f460d13942","Type":"ContainerDied","Data":"a54fed23c900166bc8331062905b6637f81535c129b6ec3f323f0d38ab66ff1e"} Oct 29 23:06:43.596298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:43.596103 4821 scope.go:122] "RemoveContainer" containerID="1fca9d6c5075da2e273cc2ee5413fc5a39d2cda980e331b44987cdf1d6785045" Oct 29 23:06:43.596298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:43.596052 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv" Oct 29 23:06:43.652923 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:43.652892 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv"] Oct 29 23:06:43.665083 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:43.665056 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4f2e3a-66c96cddbd-vhhrv"] Oct 29 23:06:44.171608 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:44.171574 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e4e8184-e3d8-40ba-865d-64f460d13942" path="/var/lib/kubelet/pods/7e4e8184-e3d8-40ba-865d-64f460d13942/volumes" Oct 29 23:06:47.125167 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:47.125125 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9"] Oct 29 23:06:51.657338 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.657304 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c"] Oct 29 23:06:51.657934 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.657920 4821 cpu_manager.go:414] "containerMap: removing container" podUID="d654f3f3-8c6b-46e4-b528-0459173f9d2a" containerName="istio-proxy" Oct 29 23:06:51.657987 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.657935 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d654f3f3-8c6b-46e4-b528-0459173f9d2a" containerName="istio-proxy" Oct 29 23:06:51.657987 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.657948 4821 cpu_manager.go:414] "containerMap: removing container" podUID="7e4e8184-e3d8-40ba-865d-64f460d13942" containerName="payload-processing" Oct 29 23:06:51.657987 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.657954 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e4e8184-e3d8-40ba-865d-64f460d13942" containerName="payload-processing" Oct 29 23:06:51.657987 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.657975 4821 cpu_manager.go:414] "containerMap: removing container" podUID="324e8f24-448e-4b50-ba97-20abef961f80" containerName="revoke-keys" Oct 29 23:06:51.657987 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.657979 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="324e8f24-448e-4b50-ba97-20abef961f80" containerName="revoke-keys" Oct 29 23:06:51.658151 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.658072 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="d654f3f3-8c6b-46e4-b528-0459173f9d2a" containerName="istio-proxy" Oct 29 23:06:51.658151 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.658080 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e4e8184-e3d8-40ba-865d-64f460d13942" containerName="payload-processing" Oct 29 23:06:51.658151 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.658089 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="324e8f24-448e-4b50-ba97-20abef961f80" containerName="revoke-keys" Oct 29 23:06:51.663144 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.663123 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c" Oct 29 23:06:51.682803 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.682777 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c"] Oct 29 23:06:51.690029 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.690003 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:06:51.690225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.690014 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:06:51.775019 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.774980 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a4efc6-779b-4265-8aa4-00b7dbaa15b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c\" (UID: \"b6a4efc6-779b-4265-8aa4-00b7dbaa15b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c" Oct 29 23:06:51.775268 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.775025 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24z2d\" (UniqueName: \"kubernetes.io/projected/b6a4efc6-779b-4265-8aa4-00b7dbaa15b0-kube-api-access-24z2d\") pod \"maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c\" (UID: \"b6a4efc6-779b-4265-8aa4-00b7dbaa15b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c" Oct 29 23:06:51.775268 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.775178 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a4efc6-779b-4265-8aa4-00b7dbaa15b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c\" (UID: \"b6a4efc6-779b-4265-8aa4-00b7dbaa15b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c" Oct 29 23:06:51.877019 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.876974 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a4efc6-779b-4265-8aa4-00b7dbaa15b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c\" (UID: \"b6a4efc6-779b-4265-8aa4-00b7dbaa15b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c" Oct 29 23:06:51.877292 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.877273 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a4efc6-779b-4265-8aa4-00b7dbaa15b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c\" (UID: \"b6a4efc6-779b-4265-8aa4-00b7dbaa15b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c" Oct 29 23:06:51.877396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.877318 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24z2d\" (UniqueName: \"kubernetes.io/projected/b6a4efc6-779b-4265-8aa4-00b7dbaa15b0-kube-api-access-24z2d\") pod \"maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c\" (UID: \"b6a4efc6-779b-4265-8aa4-00b7dbaa15b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c" Oct 29 23:06:51.877977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.877924 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a4efc6-779b-4265-8aa4-00b7dbaa15b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c\" (UID: \"b6a4efc6-779b-4265-8aa4-00b7dbaa15b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c" Oct 29 23:06:51.879651 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.879630 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a4efc6-779b-4265-8aa4-00b7dbaa15b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c\" (UID: \"b6a4efc6-779b-4265-8aa4-00b7dbaa15b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c" Oct 29 23:06:51.902327 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.902292 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24z2d\" (UniqueName: \"kubernetes.io/projected/b6a4efc6-779b-4265-8aa4-00b7dbaa15b0-kube-api-access-24z2d\") pod \"maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c\" (UID: \"b6a4efc6-779b-4265-8aa4-00b7dbaa15b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c" Oct 29 23:06:51.969878 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.969796 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs"] Oct 29 23:06:51.977391 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:51.977366 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c" Oct 29 23:06:52.001351 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:52.001321 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f"] Oct 29 23:06:52.290527 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:52.290496 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c"] Oct 29 23:06:52.290912 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:06:52.290881 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a4efc6_779b_4265_8aa4_00b7dbaa15b0.slice/crio-049ba40ef259a73fbcde6cdefc7d616098a68893b1f8c74306516eda286943ed WatchSource:0}: Error finding container 049ba40ef259a73fbcde6cdefc7d616098a68893b1f8c74306516eda286943ed: Status 404 returned error can't find the container with id 049ba40ef259a73fbcde6cdefc7d616098a68893b1f8c74306516eda286943ed Oct 29 23:06:52.872935 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:52.872869 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c" event={"ID":"b6a4efc6-779b-4265-8aa4-00b7dbaa15b0","Type":"ContainerStarted","Data":"049ba40ef259a73fbcde6cdefc7d616098a68893b1f8c74306516eda286943ed"} Oct 29 23:06:53.897238 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:53.897182 4821 generic.go:356] "Generic (PLEG): container finished" podID="b6a4efc6-779b-4265-8aa4-00b7dbaa15b0" containerID="56dcaf3bbee7514ef148d2832897f5ded7bff1af54e0494dc4ffc89e97efc7a9" exitCode=0 Oct 29 23:06:53.897943 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:53.897606 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c" event={"ID":"b6a4efc6-779b-4265-8aa4-00b7dbaa15b0","Type":"ContainerDied","Data":"56dcaf3bbee7514ef148d2832897f5ded7bff1af54e0494dc4ffc89e97efc7a9"} Oct 29 23:06:53.899474 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:06:53.899432 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:06:54.319923 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:54.319801 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd"] Oct 29 23:06:54.325668 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:54.325631 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4f2e3a-tkgvd"] Oct 29 23:06:55.094795 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:55.094768 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c" Oct 29 23:06:55.177997 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:55.177901 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6a4efc6-779b-4265-8aa4-00b7dbaa15b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a4efc6-779b-4265-8aa4-00b7dbaa15b0-maas-api-openshift-ca\") pod \"b6a4efc6-779b-4265-8aa4-00b7dbaa15b0\" (UID: \"b6a4efc6-779b-4265-8aa4-00b7dbaa15b0\") " Oct 29 23:06:55.177997 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:55.177971 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6a4efc6-779b-4265-8aa4-00b7dbaa15b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a4efc6-779b-4265-8aa4-00b7dbaa15b0-maas-api-xks-ca\") pod \"b6a4efc6-779b-4265-8aa4-00b7dbaa15b0\" (UID: \"b6a4efc6-779b-4265-8aa4-00b7dbaa15b0\") " Oct 29 23:06:55.178266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:55.178059 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6a4efc6-779b-4265-8aa4-00b7dbaa15b0-kube-api-access-24z2d\" (UniqueName: \"kubernetes.io/projected/b6a4efc6-779b-4265-8aa4-00b7dbaa15b0-kube-api-access-24z2d\") pod \"b6a4efc6-779b-4265-8aa4-00b7dbaa15b0\" (UID: \"b6a4efc6-779b-4265-8aa4-00b7dbaa15b0\") " Oct 29 23:06:55.178364 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:55.178337 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a4efc6-779b-4265-8aa4-00b7dbaa15b0-maas-api-openshift-ca" pod "b6a4efc6-779b-4265-8aa4-00b7dbaa15b0" (UID: "b6a4efc6-779b-4265-8aa4-00b7dbaa15b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:55.178780 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:55.178739 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a4efc6-779b-4265-8aa4-00b7dbaa15b0-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:55.181250 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:55.181180 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a4efc6-779b-4265-8aa4-00b7dbaa15b0-kube-api-access-24z2d" pod "b6a4efc6-779b-4265-8aa4-00b7dbaa15b0" (UID: "b6a4efc6-779b-4265-8aa4-00b7dbaa15b0"). InnerVolumeSpecName "kube-api-access-24z2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:55.183066 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:55.183039 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6a4efc6-779b-4265-8aa4-00b7dbaa15b0-maas-api-xks-ca" pod "b6a4efc6-779b-4265-8aa4-00b7dbaa15b0" (UID: "b6a4efc6-779b-4265-8aa4-00b7dbaa15b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:55.280013 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:55.279977 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a4efc6-779b-4265-8aa4-00b7dbaa15b0-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:55.280013 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:55.280013 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24z2d\" (UniqueName: \"kubernetes.io/projected/b6a4efc6-779b-4265-8aa4-00b7dbaa15b0-kube-api-access-24z2d\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:55.916373 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:55.916339 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c" event={"ID":"b6a4efc6-779b-4265-8aa4-00b7dbaa15b0","Type":"ContainerDied","Data":"049ba40ef259a73fbcde6cdefc7d616098a68893b1f8c74306516eda286943ed"} Oct 29 23:06:55.916557 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:55.916397 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="049ba40ef259a73fbcde6cdefc7d616098a68893b1f8c74306516eda286943ed" Oct 29 23:06:55.916557 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:55.916354 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c" Oct 29 23:06:56.172828 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.172740 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="324e8f24-448e-4b50-ba97-20abef961f80" path="/var/lib/kubelet/pods/324e8f24-448e-4b50-ba97-20abef961f80/volumes" Oct 29 23:06:56.575010 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.574916 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4"] Oct 29 23:06:56.575585 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.575563 4821 cpu_manager.go:414] "containerMap: removing container" podUID="b6a4efc6-779b-4265-8aa4-00b7dbaa15b0" containerName="revoke-keys" Oct 29 23:06:56.575585 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.575585 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6a4efc6-779b-4265-8aa4-00b7dbaa15b0" containerName="revoke-keys" Oct 29 23:06:56.575760 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.575611 4821 cpu_manager.go:414] "containerMap: removing container" podUID="d654f3f3-8c6b-46e4-b528-0459173f9d2a" containerName="istio-proxy" Oct 29 23:06:56.575760 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.575618 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d654f3f3-8c6b-46e4-b528-0459173f9d2a" containerName="istio-proxy" Oct 29 23:06:56.575760 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.575737 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6a4efc6-779b-4265-8aa4-00b7dbaa15b0" containerName="revoke-keys" Oct 29 23:06:56.575760 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.575747 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="d654f3f3-8c6b-46e4-b528-0459173f9d2a" containerName="istio-proxy" Oct 29 23:06:56.579772 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.579747 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.600738 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.600710 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istiod-ca-cert\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.600877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.600758 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-workload-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.600877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.600861 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j24gx\" (UniqueName: \"kubernetes.io/projected/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-kube-api-access-j24gx\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.600989 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.600957 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-data\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.601059 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.601017 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-podinfo\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.601099 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.601077 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-token\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.601187 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.601173 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-workload-certs\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.601308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.601294 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-envoy\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.601361 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.601349 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-credential-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.610770 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.610745 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4"] Oct 29 23:06:56.646489 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.646457 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-77238f-openshift-default-dockercfg-24287\"" Oct 29 23:06:56.702653 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.702623 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-workload-certs\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.702871 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.702699 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-envoy\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.702871 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.702730 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-credential-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.702871 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.702807 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istiod-ca-cert\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.702871 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.702831 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-workload-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.702871 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.702870 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j24gx\" (UniqueName: \"kubernetes.io/projected/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-kube-api-access-j24gx\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.703150 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.702924 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-data\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.703150 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.702967 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-podinfo\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.703150 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.702997 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-token\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.703150 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.703118 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-workload-certs\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.703479 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.703445 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-data\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.703616 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.703490 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-credential-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.703697 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.703650 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-workload-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.704001 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.703981 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istiod-ca-cert\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.706478 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.706455 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-podinfo\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.712465 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.712442 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-envoy\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.749829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.749798 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-token\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.750096 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.750073 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j24gx\" (UniqueName: \"kubernetes.io/projected/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-kube-api-access-j24gx\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.902766 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.902731 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:56.970465 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:56.970395 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs" podUID="c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6" containerName="payload-processing" containerID="cri-o://9f2bf3c8769f0d7402349032a98d7f42b540ae0513eff052f637b5885b50b50e" gracePeriod=25 Oct 29 23:06:57.002457 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.002413 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f" podUID="59da5ac0-2bf1-4405-8a59-35c74f5fc209" containerName="payload-pre-processing" containerID="cri-o://fc596080f0f369ab3f468e26691cd3f6688d0fb175d153a036f3862ae89373dc" gracePeriod=25 Oct 29 23:06:57.239941 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.239913 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4"] Oct 29 23:06:57.240387 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:06:57.240087 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0b8707_7668_4b6b_8a0b_4ad8fd956407.slice/crio-e161fdd9c1784de89a86eab6d8e1097f7aae9f85a3b1368471780750bced7670 WatchSource:0}: Error finding container e161fdd9c1784de89a86eab6d8e1097f7aae9f85a3b1368471780750bced7670: Status 404 returned error can't find the container with id e161fdd9c1784de89a86eab6d8e1097f7aae9f85a3b1368471780750bced7670 Oct 29 23:06:57.244651 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.244618 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:06:57.244767 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.244694 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:06:57.244767 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.244742 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:06:57.286378 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.286018 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs" Oct 29 23:06:57.312699 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.312670 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6-plugins-config-volume\") pod \"c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6\" (UID: \"c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6\") " Oct 29 23:06:57.312903 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.312771 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6-kube-api-access-2hxfl\" (UniqueName: \"kubernetes.io/projected/c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6-kube-api-access-2hxfl\") pod \"c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6\" (UID: \"c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6\") " Oct 29 23:06:57.313078 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.313052 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6-plugins-config-volume" pod "c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6" (UID: "c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:57.313532 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.313510 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:57.315653 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.315631 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6-kube-api-access-2hxfl" pod "c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6" (UID: "c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6"). InnerVolumeSpecName "kube-api-access-2hxfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:57.333503 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.333480 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f" Oct 29 23:06:57.414255 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.414045 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59da5ac0-2bf1-4405-8a59-35c74f5fc209-kube-api-access-prxzf\" (UniqueName: \"kubernetes.io/projected/59da5ac0-2bf1-4405-8a59-35c74f5fc209-kube-api-access-prxzf\") pod \"59da5ac0-2bf1-4405-8a59-35c74f5fc209\" (UID: \"59da5ac0-2bf1-4405-8a59-35c74f5fc209\") " Oct 29 23:06:57.414460 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.414378 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59da5ac0-2bf1-4405-8a59-35c74f5fc209-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59da5ac0-2bf1-4405-8a59-35c74f5fc209-plugins-config-volume\") pod \"59da5ac0-2bf1-4405-8a59-35c74f5fc209\" (UID: \"59da5ac0-2bf1-4405-8a59-35c74f5fc209\") " Oct 29 23:06:57.414990 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.414964 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hxfl\" (UniqueName: \"kubernetes.io/projected/c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6-kube-api-access-2hxfl\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:57.415308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.415273 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59da5ac0-2bf1-4405-8a59-35c74f5fc209-plugins-config-volume" pod "59da5ac0-2bf1-4405-8a59-35c74f5fc209" (UID: "59da5ac0-2bf1-4405-8a59-35c74f5fc209"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:57.420279 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.420245 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59da5ac0-2bf1-4405-8a59-35c74f5fc209-kube-api-access-prxzf" pod "59da5ac0-2bf1-4405-8a59-35c74f5fc209" (UID: "59da5ac0-2bf1-4405-8a59-35c74f5fc209"). InnerVolumeSpecName "kube-api-access-prxzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:57.516574 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.516500 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59da5ac0-2bf1-4405-8a59-35c74f5fc209-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:57.516574 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.516530 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prxzf\" (UniqueName: \"kubernetes.io/projected/59da5ac0-2bf1-4405-8a59-35c74f5fc209-kube-api-access-prxzf\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:06:57.948132 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.948097 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" event={"ID":"0c0b8707-7668-4b6b-8a0b-4ad8fd956407","Type":"ContainerStarted","Data":"92143966a1655f2d68309ed9ed4fa0dcb4f1df78d2e7ded7d4ecf2832203c8eb"} Oct 29 23:06:57.948398 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.948154 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" event={"ID":"0c0b8707-7668-4b6b-8a0b-4ad8fd956407","Type":"ContainerStarted","Data":"e161fdd9c1784de89a86eab6d8e1097f7aae9f85a3b1368471780750bced7670"} Oct 29 23:06:57.949696 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.949670 4821 generic.go:356] "Generic (PLEG): container finished" podID="59da5ac0-2bf1-4405-8a59-35c74f5fc209" containerID="fc596080f0f369ab3f468e26691cd3f6688d0fb175d153a036f3862ae89373dc" exitCode=0 Oct 29 23:06:57.949836 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.949707 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f" Oct 29 23:06:57.949836 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.949759 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f" event={"ID":"59da5ac0-2bf1-4405-8a59-35c74f5fc209","Type":"ContainerDied","Data":"fc596080f0f369ab3f468e26691cd3f6688d0fb175d153a036f3862ae89373dc"} Oct 29 23:06:57.949836 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.949813 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f" event={"ID":"59da5ac0-2bf1-4405-8a59-35c74f5fc209","Type":"ContainerDied","Data":"253cced11d064723fe04f761207b0acfc78876fe8f0f43df67dd7450ad52a888"} Oct 29 23:06:57.950007 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.949844 4821 scope.go:122] "RemoveContainer" containerID="fc596080f0f369ab3f468e26691cd3f6688d0fb175d153a036f3862ae89373dc" Oct 29 23:06:57.954099 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.954080 4821 generic.go:356] "Generic (PLEG): container finished" podID="c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6" containerID="9f2bf3c8769f0d7402349032a98d7f42b540ae0513eff052f637b5885b50b50e" exitCode=0 Oct 29 23:06:57.954192 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.954127 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs" Oct 29 23:06:57.954279 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.954174 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs" event={"ID":"c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6","Type":"ContainerDied","Data":"9f2bf3c8769f0d7402349032a98d7f42b540ae0513eff052f637b5885b50b50e"} Oct 29 23:06:57.954422 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.954404 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs" event={"ID":"c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6","Type":"ContainerDied","Data":"fb512cd031f8ec3dad6872c848463d26f51b2e5b623a017614b7dc6ea711096a"} Oct 29 23:06:57.964610 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.964590 4821 scope.go:122] "RemoveContainer" containerID="fc596080f0f369ab3f468e26691cd3f6688d0fb175d153a036f3862ae89373dc" Oct 29 23:06:57.964873 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:06:57.964852 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc596080f0f369ab3f468e26691cd3f6688d0fb175d153a036f3862ae89373dc\": container with ID starting with fc596080f0f369ab3f468e26691cd3f6688d0fb175d153a036f3862ae89373dc not found: ID does not exist" containerID="fc596080f0f369ab3f468e26691cd3f6688d0fb175d153a036f3862ae89373dc" Oct 29 23:06:57.964923 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.964883 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc596080f0f369ab3f468e26691cd3f6688d0fb175d153a036f3862ae89373dc"} err="failed to get container status \"fc596080f0f369ab3f468e26691cd3f6688d0fb175d153a036f3862ae89373dc\": rpc error: code = NotFound desc = could not find container \"fc596080f0f369ab3f468e26691cd3f6688d0fb175d153a036f3862ae89373dc\": container with ID starting with fc596080f0f369ab3f468e26691cd3f6688d0fb175d153a036f3862ae89373dc not found: ID does not exist" Oct 29 23:06:57.964923 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.964900 4821 scope.go:122] "RemoveContainer" containerID="9f2bf3c8769f0d7402349032a98d7f42b540ae0513eff052f637b5885b50b50e" Oct 29 23:06:57.975588 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.975572 4821 scope.go:122] "RemoveContainer" containerID="9f2bf3c8769f0d7402349032a98d7f42b540ae0513eff052f637b5885b50b50e" Oct 29 23:06:57.975856 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:06:57.975833 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f2bf3c8769f0d7402349032a98d7f42b540ae0513eff052f637b5885b50b50e\": container with ID starting with 9f2bf3c8769f0d7402349032a98d7f42b540ae0513eff052f637b5885b50b50e not found: ID does not exist" containerID="9f2bf3c8769f0d7402349032a98d7f42b540ae0513eff052f637b5885b50b50e" Oct 29 23:06:57.975942 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:57.975856 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f2bf3c8769f0d7402349032a98d7f42b540ae0513eff052f637b5885b50b50e"} err="failed to get container status \"9f2bf3c8769f0d7402349032a98d7f42b540ae0513eff052f637b5885b50b50e\": rpc error: code = NotFound desc = could not find container \"9f2bf3c8769f0d7402349032a98d7f42b540ae0513eff052f637b5885b50b50e\": container with ID starting with 9f2bf3c8769f0d7402349032a98d7f42b540ae0513eff052f637b5885b50b50e not found: ID does not exist" Oct 29 23:06:58.039531 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:58.039456 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" podStartSLOduration=2.039437658 podStartE2EDuration="2.039437658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:58.00818763 +0000 UTC m=+1471.948735542" watchObservedRunningTime="2026-10-29 23:06:58.039437658 +0000 UTC m=+1471.979985568" Oct 29 23:06:58.065867 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:58.065832 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f"] Oct 29 23:06:58.072060 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:58.072031 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-77238f-685b7cbf7f-q2v6f"] Oct 29 23:06:58.107041 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:58.107014 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs"] Oct 29 23:06:58.111878 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:58.111851 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77238f-8658796568-j2xvs"] Oct 29 23:06:58.171069 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:58.171036 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59da5ac0-2bf1-4405-8a59-35c74f5fc209" path="/var/lib/kubelet/pods/59da5ac0-2bf1-4405-8a59-35c74f5fc209/volumes" Oct 29 23:06:58.171541 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:58.171521 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6" path="/var/lib/kubelet/pods/c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6/volumes" Oct 29 23:06:58.903537 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:58.903496 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:06:58.905123 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:58.905087 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 29 23:06:58.905326 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:58.905152 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" podUID="0c0b8707-7668-4b6b-8a0b-4ad8fd956407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 29 23:06:59.903886 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:59.903845 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 29 23:06:59.904428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:06:59.903918 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" podUID="0c0b8707-7668-4b6b-8a0b-4ad8fd956407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 29 23:07:00.903430 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:00.903346 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 29 23:07:00.903430 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:00.903417 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" podUID="0c0b8707-7668-4b6b-8a0b-4ad8fd956407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 29 23:07:01.903393 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:01.903365 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 29 23:07:01.903814 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:01.903415 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" podUID="0c0b8707-7668-4b6b-8a0b-4ad8fd956407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 29 23:07:02.155894 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:02.155807 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" podUID="d0417ea4-6fa4-4ab8-8087-634347f03002" containerName="main" containerID="cri-o://cbb541155e8a90212889ed3a60c3c7ecb2e9c49f09977a8b15c25cbf24a8da00" gracePeriod=45 Oct 29 23:07:02.244932 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:02.244891 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" podUID="d0417ea4-6fa4-4ab8-8087-634347f03002" containerName="main" probeResult="failure" output="Get \"https://10.133.0.149:8000/ready\": dial tcp 10.133.0.149:8000: connect: connection refused" Oct 29 23:07:02.653652 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:02.653616 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:07:02.768086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:02.765292 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-tmp-dir\") pod \"d0417ea4-6fa4-4ab8-8087-634347f03002\" (UID: \"d0417ea4-6fa4-4ab8-8087-634347f03002\") " Oct 29 23:07:02.768086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:02.765588 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-tmp-dir" pod "d0417ea4-6fa4-4ab8-8087-634347f03002" (UID: "d0417ea4-6fa4-4ab8-8087-634347f03002"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:02.768086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:02.765756 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-model-cache\") pod \"d0417ea4-6fa4-4ab8-8087-634347f03002\" (UID: \"d0417ea4-6fa4-4ab8-8087-634347f03002\") " Oct 29 23:07:02.768086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:02.765845 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-home\" (UniqueName: \"kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-home\") pod \"d0417ea4-6fa4-4ab8-8087-634347f03002\" (UID: \"d0417ea4-6fa4-4ab8-8087-634347f03002\") " Oct 29 23:07:02.768086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:02.765915 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0417ea4-6fa4-4ab8-8087-634347f03002-kube-api-access-4hfmq\" (UniqueName: \"kubernetes.io/projected/d0417ea4-6fa4-4ab8-8087-634347f03002-kube-api-access-4hfmq\") pod \"d0417ea4-6fa4-4ab8-8087-634347f03002\" (UID: \"d0417ea4-6fa4-4ab8-8087-634347f03002\") " Oct 29 23:07:02.768086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:02.765977 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-dshm\") pod \"d0417ea4-6fa4-4ab8-8087-634347f03002\" (UID: \"d0417ea4-6fa4-4ab8-8087-634347f03002\") " Oct 29 23:07:02.768086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:02.766056 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0417ea4-6fa4-4ab8-8087-634347f03002-tls-certs\" (UniqueName: \"kubernetes.io/secret/d0417ea4-6fa4-4ab8-8087-634347f03002-tls-certs\") pod \"d0417ea4-6fa4-4ab8-8087-634347f03002\" (UID: \"d0417ea4-6fa4-4ab8-8087-634347f03002\") " Oct 29 23:07:02.768086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:02.766844 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:02.768086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:02.767298 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-home" pod "d0417ea4-6fa4-4ab8-8087-634347f03002" (UID: "d0417ea4-6fa4-4ab8-8087-634347f03002"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:02.768086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:02.767478 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-model-cache" pod "d0417ea4-6fa4-4ab8-8087-634347f03002" (UID: "d0417ea4-6fa4-4ab8-8087-634347f03002"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:02.769853 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:02.769828 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0417ea4-6fa4-4ab8-8087-634347f03002-tls-certs" pod "d0417ea4-6fa4-4ab8-8087-634347f03002" (UID: "d0417ea4-6fa4-4ab8-8087-634347f03002"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:02.770144 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:02.770104 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-dshm" pod "d0417ea4-6fa4-4ab8-8087-634347f03002" (UID: "d0417ea4-6fa4-4ab8-8087-634347f03002"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:02.770860 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:02.770837 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0417ea4-6fa4-4ab8-8087-634347f03002-kube-api-access-4hfmq" pod "d0417ea4-6fa4-4ab8-8087-634347f03002" (UID: "d0417ea4-6fa4-4ab8-8087-634347f03002"). InnerVolumeSpecName "kube-api-access-4hfmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:02.868386 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:02.868354 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0417ea4-6fa4-4ab8-8087-634347f03002-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:02.868614 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:02.868600 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:02.868727 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:02.868714 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:02.868839 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:02.868819 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hfmq\" (UniqueName: \"kubernetes.io/projected/d0417ea4-6fa4-4ab8-8087-634347f03002-kube-api-access-4hfmq\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:02.869274 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:02.869254 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0417ea4-6fa4-4ab8-8087-634347f03002-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:02.905336 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:02.905300 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 29 23:07:02.905963 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:02.905915 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" podUID="0c0b8707-7668-4b6b-8a0b-4ad8fd956407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 29 23:07:03.021445 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:03.021348 4821 generic.go:356] "Generic (PLEG): container finished" podID="d0417ea4-6fa4-4ab8-8087-634347f03002" containerID="cbb541155e8a90212889ed3a60c3c7ecb2e9c49f09977a8b15c25cbf24a8da00" exitCode=0 Oct 29 23:07:03.021725 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:03.021699 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" event={"ID":"d0417ea4-6fa4-4ab8-8087-634347f03002","Type":"ContainerDied","Data":"cbb541155e8a90212889ed3a60c3c7ecb2e9c49f09977a8b15c25cbf24a8da00"} Oct 29 23:07:03.021862 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:03.021846 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" event={"ID":"d0417ea4-6fa4-4ab8-8087-634347f03002","Type":"ContainerDied","Data":"ac484c6dfd6e8275bcf0a76e7dca64a0b7eae441b3ec67d54419298a8699e212"} Oct 29 23:07:03.021970 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:03.021956 4821 scope.go:122] "RemoveContainer" containerID="cbb541155e8a90212889ed3a60c3c7ecb2e9c49f09977a8b15c25cbf24a8da00" Oct 29 23:07:03.022257 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:03.022238 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9" Oct 29 23:07:03.077068 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:03.077038 4821 scope.go:122] "RemoveContainer" containerID="cbb541155e8a90212889ed3a60c3c7ecb2e9c49f09977a8b15c25cbf24a8da00" Oct 29 23:07:03.077886 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:07:03.077672 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb541155e8a90212889ed3a60c3c7ecb2e9c49f09977a8b15c25cbf24a8da00\": container with ID starting with cbb541155e8a90212889ed3a60c3c7ecb2e9c49f09977a8b15c25cbf24a8da00 not found: ID does not exist" containerID="cbb541155e8a90212889ed3a60c3c7ecb2e9c49f09977a8b15c25cbf24a8da00" Oct 29 23:07:03.077886 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:03.077724 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb541155e8a90212889ed3a60c3c7ecb2e9c49f09977a8b15c25cbf24a8da00"} err="failed to get container status \"cbb541155e8a90212889ed3a60c3c7ecb2e9c49f09977a8b15c25cbf24a8da00\": rpc error: code = NotFound desc = could not find container \"cbb541155e8a90212889ed3a60c3c7ecb2e9c49f09977a8b15c25cbf24a8da00\": container with ID starting with cbb541155e8a90212889ed3a60c3c7ecb2e9c49f09977a8b15c25cbf24a8da00 not found: ID does not exist" Oct 29 23:07:03.097700 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:03.095267 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9"] Oct 29 23:07:03.099180 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:03.099149 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-7d18f7/ipp-route-7d18f7-kserve-794d8647c-n5nv9"] Oct 29 23:07:03.908614 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:03.908583 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:07:03.909087 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:03.909065 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:07:03.909785 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:03.909767 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:07:04.173972 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:04.173884 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0417ea4-6fa4-4ab8-8087-634347f03002" path="/var/lib/kubelet/pods/d0417ea4-6fa4-4ab8-8087-634347f03002/volumes" Oct 29 23:07:08.189561 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.189529 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a57b94a-rpkhn"] Oct 29 23:07:08.190123 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.190108 4821 cpu_manager.go:414] "containerMap: removing container" podUID="c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6" containerName="payload-processing" Oct 29 23:07:08.190171 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.190124 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6" containerName="payload-processing" Oct 29 23:07:08.190171 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.190149 4821 cpu_manager.go:414] "containerMap: removing container" podUID="59da5ac0-2bf1-4405-8a59-35c74f5fc209" containerName="payload-pre-processing" Oct 29 23:07:08.190171 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.190154 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59da5ac0-2bf1-4405-8a59-35c74f5fc209" containerName="payload-pre-processing" Oct 29 23:07:08.190171 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.190168 4821 cpu_manager.go:414] "containerMap: removing container" podUID="d0417ea4-6fa4-4ab8-8087-634347f03002" containerName="main" Oct 29 23:07:08.190306 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.190173 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0417ea4-6fa4-4ab8-8087-634347f03002" containerName="main" Oct 29 23:07:08.190306 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.190291 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="59da5ac0-2bf1-4405-8a59-35c74f5fc209" containerName="payload-pre-processing" Oct 29 23:07:08.190306 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.190300 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0417ea4-6fa4-4ab8-8087-634347f03002" containerName="main" Oct 29 23:07:08.190401 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.190313 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="c56e5474-3d55-4c4a-a3f9-c137f7c2d0d6" containerName="payload-processing" Oct 29 23:07:08.195186 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.195169 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a57b94a-rpkhn" Oct 29 23:07:08.237634 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.237601 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a57b94a-rpkhn"] Oct 29 23:07:08.245102 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.245076 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:07:08.245258 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.245150 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:07:08.251849 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.251808 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eebc6c7-70cb-4417-8db8-d6e0530b7503-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a57b94a-rpkhn\" (UID: \"5eebc6c7-70cb-4417-8db8-d6e0530b7503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a57b94a-rpkhn" Oct 29 23:07:08.251978 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.251873 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w8g9\" (UniqueName: \"kubernetes.io/projected/5eebc6c7-70cb-4417-8db8-d6e0530b7503-kube-api-access-6w8g9\") pod \"maas-api-revoke-sub-8a57b94a-rpkhn\" (UID: \"5eebc6c7-70cb-4417-8db8-d6e0530b7503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a57b94a-rpkhn" Oct 29 23:07:08.252123 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.252099 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eebc6c7-70cb-4417-8db8-d6e0530b7503-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a57b94a-rpkhn\" (UID: \"5eebc6c7-70cb-4417-8db8-d6e0530b7503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a57b94a-rpkhn" Oct 29 23:07:08.353534 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.353498 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eebc6c7-70cb-4417-8db8-d6e0530b7503-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a57b94a-rpkhn\" (UID: \"5eebc6c7-70cb-4417-8db8-d6e0530b7503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a57b94a-rpkhn" Oct 29 23:07:08.353769 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.353750 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eebc6c7-70cb-4417-8db8-d6e0530b7503-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a57b94a-rpkhn\" (UID: \"5eebc6c7-70cb-4417-8db8-d6e0530b7503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a57b94a-rpkhn" Oct 29 23:07:08.353832 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.353811 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w8g9\" (UniqueName: \"kubernetes.io/projected/5eebc6c7-70cb-4417-8db8-d6e0530b7503-kube-api-access-6w8g9\") pod \"maas-api-revoke-sub-8a57b94a-rpkhn\" (UID: \"5eebc6c7-70cb-4417-8db8-d6e0530b7503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a57b94a-rpkhn" Oct 29 23:07:08.354420 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.354394 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eebc6c7-70cb-4417-8db8-d6e0530b7503-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a57b94a-rpkhn\" (UID: \"5eebc6c7-70cb-4417-8db8-d6e0530b7503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a57b94a-rpkhn" Oct 29 23:07:08.356024 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.356006 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eebc6c7-70cb-4417-8db8-d6e0530b7503-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a57b94a-rpkhn\" (UID: \"5eebc6c7-70cb-4417-8db8-d6e0530b7503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a57b94a-rpkhn" Oct 29 23:07:08.380387 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.380363 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w8g9\" (UniqueName: \"kubernetes.io/projected/5eebc6c7-70cb-4417-8db8-d6e0530b7503-kube-api-access-6w8g9\") pod \"maas-api-revoke-sub-8a57b94a-rpkhn\" (UID: \"5eebc6c7-70cb-4417-8db8-d6e0530b7503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a57b94a-rpkhn" Oct 29 23:07:08.508895 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.508818 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a57b94a-rpkhn" Oct 29 23:07:08.820066 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:08.820036 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a57b94a-rpkhn"] Oct 29 23:07:08.820494 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:07:08.820470 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eebc6c7_70cb_4417_8db8_d6e0530b7503.slice/crio-10dbfc14c5d44e7bc4faf0ab37d7a078be180eb3cf42599473f5d0bbf26f5f39 WatchSource:0}: Error finding container 10dbfc14c5d44e7bc4faf0ab37d7a078be180eb3cf42599473f5d0bbf26f5f39: Status 404 returned error can't find the container with id 10dbfc14c5d44e7bc4faf0ab37d7a078be180eb3cf42599473f5d0bbf26f5f39 Oct 29 23:07:09.081173 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:09.081091 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a57b94a-rpkhn" event={"ID":"5eebc6c7-70cb-4417-8db8-d6e0530b7503","Type":"ContainerStarted","Data":"10dbfc14c5d44e7bc4faf0ab37d7a078be180eb3cf42599473f5d0bbf26f5f39"} Oct 29 23:07:10.091550 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:10.091456 4821 generic.go:356] "Generic (PLEG): container finished" podID="5eebc6c7-70cb-4417-8db8-d6e0530b7503" containerID="fd1c3bcc5eeab13db4e5d666685e8a41cebcd26389afe4602e85cfd0e3f4a743" exitCode=0 Oct 29 23:07:10.091550 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:10.091513 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a57b94a-rpkhn" event={"ID":"5eebc6c7-70cb-4417-8db8-d6e0530b7503","Type":"ContainerDied","Data":"fd1c3bcc5eeab13db4e5d666685e8a41cebcd26389afe4602e85cfd0e3f4a743"} Oct 29 23:07:11.292986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:11.292956 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a57b94a-rpkhn" Oct 29 23:07:11.408502 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:11.408468 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eebc6c7-70cb-4417-8db8-d6e0530b7503-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eebc6c7-70cb-4417-8db8-d6e0530b7503-maas-api-openshift-ca\") pod \"5eebc6c7-70cb-4417-8db8-d6e0530b7503\" (UID: \"5eebc6c7-70cb-4417-8db8-d6e0530b7503\") " Oct 29 23:07:11.408710 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:11.408576 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eebc6c7-70cb-4417-8db8-d6e0530b7503-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eebc6c7-70cb-4417-8db8-d6e0530b7503-maas-api-xks-ca\") pod \"5eebc6c7-70cb-4417-8db8-d6e0530b7503\" (UID: \"5eebc6c7-70cb-4417-8db8-d6e0530b7503\") " Oct 29 23:07:11.408782 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:11.408748 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eebc6c7-70cb-4417-8db8-d6e0530b7503-kube-api-access-6w8g9\" (UniqueName: \"kubernetes.io/projected/5eebc6c7-70cb-4417-8db8-d6e0530b7503-kube-api-access-6w8g9\") pod \"5eebc6c7-70cb-4417-8db8-d6e0530b7503\" (UID: \"5eebc6c7-70cb-4417-8db8-d6e0530b7503\") " Oct 29 23:07:11.408869 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:11.408842 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eebc6c7-70cb-4417-8db8-d6e0530b7503-maas-api-openshift-ca" pod "5eebc6c7-70cb-4417-8db8-d6e0530b7503" (UID: "5eebc6c7-70cb-4417-8db8-d6e0530b7503"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:11.409508 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:11.409484 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eebc6c7-70cb-4417-8db8-d6e0530b7503-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:11.410963 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:11.410934 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eebc6c7-70cb-4417-8db8-d6e0530b7503-maas-api-xks-ca" pod "5eebc6c7-70cb-4417-8db8-d6e0530b7503" (UID: "5eebc6c7-70cb-4417-8db8-d6e0530b7503"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:11.411075 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:11.411053 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eebc6c7-70cb-4417-8db8-d6e0530b7503-kube-api-access-6w8g9" pod "5eebc6c7-70cb-4417-8db8-d6e0530b7503" (UID: "5eebc6c7-70cb-4417-8db8-d6e0530b7503"). InnerVolumeSpecName "kube-api-access-6w8g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:11.510561 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:11.510529 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w8g9\" (UniqueName: \"kubernetes.io/projected/5eebc6c7-70cb-4417-8db8-d6e0530b7503-kube-api-access-6w8g9\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:11.510561 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:11.510556 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eebc6c7-70cb-4417-8db8-d6e0530b7503-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:12.108501 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:12.108474 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a57b94a-rpkhn" Oct 29 23:07:12.108663 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:12.108506 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a57b94a-rpkhn" event={"ID":"5eebc6c7-70cb-4417-8db8-d6e0530b7503","Type":"ContainerDied","Data":"10dbfc14c5d44e7bc4faf0ab37d7a078be180eb3cf42599473f5d0bbf26f5f39"} Oct 29 23:07:12.108663 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:12.108575 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10dbfc14c5d44e7bc4faf0ab37d7a078be180eb3cf42599473f5d0bbf26f5f39" Oct 29 23:07:18.942997 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:18.942951 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd"] Oct 29 23:07:18.943698 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:18.943679 4821 cpu_manager.go:414] "containerMap: removing container" podUID="5eebc6c7-70cb-4417-8db8-d6e0530b7503" containerName="revoke-keys" Oct 29 23:07:18.943761 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:18.943700 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eebc6c7-70cb-4417-8db8-d6e0530b7503" containerName="revoke-keys" Oct 29 23:07:18.943840 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:18.943830 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eebc6c7-70cb-4417-8db8-d6e0530b7503" containerName="revoke-keys" Oct 29 23:07:18.947535 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:18.947516 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd" Oct 29 23:07:18.965102 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:18.965077 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd"] Oct 29 23:07:19.039541 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:19.039479 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:07:19.039742 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:19.039602 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:07:19.042134 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:19.042105 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be298eab-1385-494b-b0a1-97930eede6bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd\" (UID: \"be298eab-1385-494b-b0a1-97930eede6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd" Oct 29 23:07:19.042392 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:19.042366 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdbsk\" (UniqueName: \"kubernetes.io/projected/be298eab-1385-494b-b0a1-97930eede6bd-kube-api-access-xdbsk\") pod \"maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd\" (UID: \"be298eab-1385-494b-b0a1-97930eede6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd" Oct 29 23:07:19.042528 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:19.042512 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be298eab-1385-494b-b0a1-97930eede6bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd\" (UID: \"be298eab-1385-494b-b0a1-97930eede6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd" Oct 29 23:07:19.144292 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:19.144248 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be298eab-1385-494b-b0a1-97930eede6bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd\" (UID: \"be298eab-1385-494b-b0a1-97930eede6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd" Oct 29 23:07:19.144465 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:19.144443 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdbsk\" (UniqueName: \"kubernetes.io/projected/be298eab-1385-494b-b0a1-97930eede6bd-kube-api-access-xdbsk\") pod \"maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd\" (UID: \"be298eab-1385-494b-b0a1-97930eede6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd" Oct 29 23:07:19.144576 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:19.144562 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be298eab-1385-494b-b0a1-97930eede6bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd\" (UID: \"be298eab-1385-494b-b0a1-97930eede6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd" Oct 29 23:07:19.144936 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:19.144912 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be298eab-1385-494b-b0a1-97930eede6bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd\" (UID: \"be298eab-1385-494b-b0a1-97930eede6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd" Oct 29 23:07:19.147032 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:19.147009 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be298eab-1385-494b-b0a1-97930eede6bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd\" (UID: \"be298eab-1385-494b-b0a1-97930eede6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd" Oct 29 23:07:19.174081 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:19.174056 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdbsk\" (UniqueName: \"kubernetes.io/projected/be298eab-1385-494b-b0a1-97930eede6bd-kube-api-access-xdbsk\") pod \"maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd\" (UID: \"be298eab-1385-494b-b0a1-97930eede6bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd" Oct 29 23:07:19.263321 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:19.263241 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd" Oct 29 23:07:19.647260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:19.647076 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd"] Oct 29 23:07:20.201369 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:20.201334 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd" event={"ID":"be298eab-1385-494b-b0a1-97930eede6bd","Type":"ContainerStarted","Data":"c6b8f16f620933cacc22778a070f3b5c8865f405c31857611d2b52033d5092ff"} Oct 29 23:07:20.201741 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:20.201613 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd" event={"ID":"be298eab-1385-494b-b0a1-97930eede6bd","Type":"ContainerStarted","Data":"e2ec87accab556198c22b2c27f6cd8746ea74dcc4a1adfed700ca1c1f211233d"} Oct 29 23:07:20.292234 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:20.292159 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd" podStartSLOduration=2.292145934 podStartE2EDuration="2.292145934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:20.288252729 +0000 UTC m=+1494.228800641" watchObservedRunningTime="2026-10-29 23:07:20.292145934 +0000 UTC m=+1494.232693844" Oct 29 23:07:21.213433 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:21.213399 4821 generic.go:356] "Generic (PLEG): container finished" podID="be298eab-1385-494b-b0a1-97930eede6bd" containerID="c6b8f16f620933cacc22778a070f3b5c8865f405c31857611d2b52033d5092ff" exitCode=0 Oct 29 23:07:21.213433 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:21.213436 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd" event={"ID":"be298eab-1385-494b-b0a1-97930eede6bd","Type":"ContainerDied","Data":"c6b8f16f620933cacc22778a070f3b5c8865f405c31857611d2b52033d5092ff"} Oct 29 23:07:21.913763 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:21.913722 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30791c5-jjf5n"] Oct 29 23:07:21.918319 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:21.918293 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30791c5-jjf5n" Oct 29 23:07:21.938664 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:21.938636 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30791c5-jjf5n"] Oct 29 23:07:22.003260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:22.003217 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fb36af1-9a9c-45fc-9996-ffa8bfd7d690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b30791c5-jjf5n\" (UID: \"1fb36af1-9a9c-45fc-9996-ffa8bfd7d690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30791c5-jjf5n" Oct 29 23:07:22.003434 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:22.003395 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fb36af1-9a9c-45fc-9996-ffa8bfd7d690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b30791c5-jjf5n\" (UID: \"1fb36af1-9a9c-45fc-9996-ffa8bfd7d690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30791c5-jjf5n" Oct 29 23:07:22.003497 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:22.003439 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q69jm\" (UniqueName: \"kubernetes.io/projected/1fb36af1-9a9c-45fc-9996-ffa8bfd7d690-kube-api-access-q69jm\") pod \"maas-api-revoke-sub-b30791c5-jjf5n\" (UID: \"1fb36af1-9a9c-45fc-9996-ffa8bfd7d690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30791c5-jjf5n" Oct 29 23:07:22.105020 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:22.104987 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fb36af1-9a9c-45fc-9996-ffa8bfd7d690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b30791c5-jjf5n\" (UID: \"1fb36af1-9a9c-45fc-9996-ffa8bfd7d690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30791c5-jjf5n" Oct 29 23:07:22.105180 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:22.105154 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fb36af1-9a9c-45fc-9996-ffa8bfd7d690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b30791c5-jjf5n\" (UID: \"1fb36af1-9a9c-45fc-9996-ffa8bfd7d690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30791c5-jjf5n" Oct 29 23:07:22.105285 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:22.105192 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q69jm\" (UniqueName: \"kubernetes.io/projected/1fb36af1-9a9c-45fc-9996-ffa8bfd7d690-kube-api-access-q69jm\") pod \"maas-api-revoke-sub-b30791c5-jjf5n\" (UID: \"1fb36af1-9a9c-45fc-9996-ffa8bfd7d690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30791c5-jjf5n" Oct 29 23:07:22.106009 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:22.105979 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fb36af1-9a9c-45fc-9996-ffa8bfd7d690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b30791c5-jjf5n\" (UID: \"1fb36af1-9a9c-45fc-9996-ffa8bfd7d690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30791c5-jjf5n" Oct 29 23:07:22.107652 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:22.107631 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fb36af1-9a9c-45fc-9996-ffa8bfd7d690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b30791c5-jjf5n\" (UID: \"1fb36af1-9a9c-45fc-9996-ffa8bfd7d690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30791c5-jjf5n" Oct 29 23:07:22.130147 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:22.130110 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q69jm\" (UniqueName: \"kubernetes.io/projected/1fb36af1-9a9c-45fc-9996-ffa8bfd7d690-kube-api-access-q69jm\") pod \"maas-api-revoke-sub-b30791c5-jjf5n\" (UID: \"1fb36af1-9a9c-45fc-9996-ffa8bfd7d690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30791c5-jjf5n" Oct 29 23:07:22.236104 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:22.236071 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30791c5-jjf5n" Oct 29 23:07:22.425410 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:22.425379 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd" Oct 29 23:07:22.512826 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:22.512735 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be298eab-1385-494b-b0a1-97930eede6bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be298eab-1385-494b-b0a1-97930eede6bd-maas-api-openshift-ca\") pod \"be298eab-1385-494b-b0a1-97930eede6bd\" (UID: \"be298eab-1385-494b-b0a1-97930eede6bd\") " Oct 29 23:07:22.513016 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:22.512833 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be298eab-1385-494b-b0a1-97930eede6bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be298eab-1385-494b-b0a1-97930eede6bd-maas-api-xks-ca\") pod \"be298eab-1385-494b-b0a1-97930eede6bd\" (UID: \"be298eab-1385-494b-b0a1-97930eede6bd\") " Oct 29 23:07:22.513160 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:22.513139 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be298eab-1385-494b-b0a1-97930eede6bd-kube-api-access-xdbsk\" (UniqueName: \"kubernetes.io/projected/be298eab-1385-494b-b0a1-97930eede6bd-kube-api-access-xdbsk\") pod \"be298eab-1385-494b-b0a1-97930eede6bd\" (UID: \"be298eab-1385-494b-b0a1-97930eede6bd\") " Oct 29 23:07:22.513160 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:22.513150 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be298eab-1385-494b-b0a1-97930eede6bd-maas-api-openshift-ca" pod "be298eab-1385-494b-b0a1-97930eede6bd" (UID: "be298eab-1385-494b-b0a1-97930eede6bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:22.514179 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:22.514153 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be298eab-1385-494b-b0a1-97930eede6bd-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:22.515805 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:22.515763 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be298eab-1385-494b-b0a1-97930eede6bd-kube-api-access-xdbsk" pod "be298eab-1385-494b-b0a1-97930eede6bd" (UID: "be298eab-1385-494b-b0a1-97930eede6bd"). InnerVolumeSpecName "kube-api-access-xdbsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:22.515927 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:22.515858 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be298eab-1385-494b-b0a1-97930eede6bd-maas-api-xks-ca" pod "be298eab-1385-494b-b0a1-97930eede6bd" (UID: "be298eab-1385-494b-b0a1-97930eede6bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:22.616560 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:22.616524 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdbsk\" (UniqueName: \"kubernetes.io/projected/be298eab-1385-494b-b0a1-97930eede6bd-kube-api-access-xdbsk\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:22.616560 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:22.616553 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be298eab-1385-494b-b0a1-97930eede6bd-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:22.624028 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:22.623993 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30791c5-jjf5n"] Oct 29 23:07:22.624982 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:07:22.624937 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb36af1_9a9c_45fc_9996_ffa8bfd7d690.slice/crio-6cd7ebbfcc38dfed1c32b343466ec44a4b654c1ac54cf5963bbe02f0c3b9b6c1 WatchSource:0}: Error finding container 6cd7ebbfcc38dfed1c32b343466ec44a4b654c1ac54cf5963bbe02f0c3b9b6c1: Status 404 returned error can't find the container with id 6cd7ebbfcc38dfed1c32b343466ec44a4b654c1ac54cf5963bbe02f0c3b9b6c1 Oct 29 23:07:23.063404 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.063371 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s"] Oct 29 23:07:23.064074 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.064058 4821 cpu_manager.go:414] "containerMap: removing container" podUID="be298eab-1385-494b-b0a1-97930eede6bd" containerName="revoke-keys" Oct 29 23:07:23.064135 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.064075 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be298eab-1385-494b-b0a1-97930eede6bd" containerName="revoke-keys" Oct 29 23:07:23.064181 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.064172 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="be298eab-1385-494b-b0a1-97930eede6bd" containerName="revoke-keys" Oct 29 23:07:23.076575 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.076550 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.096313 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.095973 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s"] Oct 29 23:07:23.128694 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.128281 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zmf2\" (UniqueName: \"kubernetes.io/projected/041cb60f-5e7c-47d7-8e83-cce35d4c9529-kube-api-access-2zmf2\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.128694 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.128344 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.128694 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.128406 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.128694 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.128443 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.128694 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.128491 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.128694 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.128516 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.128694 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.128573 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.128694 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.128694 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.129266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.128906 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.230254 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.230213 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.230497 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.230320 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zmf2\" (UniqueName: \"kubernetes.io/projected/041cb60f-5e7c-47d7-8e83-cce35d4c9529-kube-api-access-2zmf2\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.230497 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.230356 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.230497 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.230408 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.230497 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.230444 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.230497 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.230489 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.230826 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.230515 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.230826 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.230576 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.230826 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.230703 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.231000 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.230832 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-data\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.231000 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.230967 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-credential-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.231133 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.231099 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-workload-socket\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.231318 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.231289 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-workload-certs\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.231842 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.231812 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istiod-ca-cert\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.233444 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.233416 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-envoy\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.233632 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.233614 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-podinfo\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.237806 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.237779 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30791c5-jjf5n" event={"ID":"1fb36af1-9a9c-45fc-9996-ffa8bfd7d690","Type":"ContainerStarted","Data":"b8f3ada00cb1300877dfff594f7b571b8147e10d7232bffbf03f08678ff785df"} Oct 29 23:07:23.238243 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.237834 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30791c5-jjf5n" event={"ID":"1fb36af1-9a9c-45fc-9996-ffa8bfd7d690","Type":"ContainerStarted","Data":"6cd7ebbfcc38dfed1c32b343466ec44a4b654c1ac54cf5963bbe02f0c3b9b6c1"} Oct 29 23:07:23.239487 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.239455 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd" event={"ID":"be298eab-1385-494b-b0a1-97930eede6bd","Type":"ContainerDied","Data":"e2ec87accab556198c22b2c27f6cd8746ea74dcc4a1adfed700ca1c1f211233d"} Oct 29 23:07:23.239605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.239507 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2ec87accab556198c22b2c27f6cd8746ea74dcc4a1adfed700ca1c1f211233d" Oct 29 23:07:23.239605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.239474 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd" Oct 29 23:07:23.257305 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.257279 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-token\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.257474 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.257418 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zmf2\" (UniqueName: \"kubernetes.io/projected/041cb60f-5e7c-47d7-8e83-cce35d4c9529-kube-api-access-2zmf2\") pod \"e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.282474 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.282423 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30791c5-jjf5n" podStartSLOduration=2.282408707 podStartE2EDuration="2.282408707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:23.2814422 +0000 UTC m=+1497.221990110" watchObservedRunningTime="2026-10-29 23:07:23.282408707 +0000 UTC m=+1497.222956616" Oct 29 23:07:23.411422 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.411390 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:23.766897 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.766868 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s"] Oct 29 23:07:23.767179 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:07:23.767155 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041cb60f_5e7c_47d7_8e83_cce35d4c9529.slice/crio-49e9848d0d000a6d4d2d0b9546a180fbcd19caebbc2d490c9bbbfec26624e0e8 WatchSource:0}: Error finding container 49e9848d0d000a6d4d2d0b9546a180fbcd19caebbc2d490c9bbbfec26624e0e8: Status 404 returned error can't find the container with id 49e9848d0d000a6d4d2d0b9546a180fbcd19caebbc2d490c9bbbfec26624e0e8 Oct 29 23:07:23.770265 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.770224 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:07:23.770723 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.770292 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:07:23.770723 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:23.770324 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:07:23.972321 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:07:23.972129 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:07:24.259764 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:24.259118 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" event={"ID":"041cb60f-5e7c-47d7-8e83-cce35d4c9529","Type":"ContainerStarted","Data":"927a14a66db06c1f045de17016d17f867b327016462e4f2f84519c9643b62ac4"} Oct 29 23:07:24.259764 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:24.259215 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" event={"ID":"041cb60f-5e7c-47d7-8e83-cce35d4c9529","Type":"ContainerStarted","Data":"49e9848d0d000a6d4d2d0b9546a180fbcd19caebbc2d490c9bbbfec26624e0e8"} Oct 29 23:07:24.263025 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:24.262993 4821 generic.go:356] "Generic (PLEG): container finished" podID="1fb36af1-9a9c-45fc-9996-ffa8bfd7d690" containerID="b8f3ada00cb1300877dfff594f7b571b8147e10d7232bffbf03f08678ff785df" exitCode=0 Oct 29 23:07:24.263170 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:24.263128 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30791c5-jjf5n" event={"ID":"1fb36af1-9a9c-45fc-9996-ffa8bfd7d690","Type":"ContainerDied","Data":"b8f3ada00cb1300877dfff594f7b571b8147e10d7232bffbf03f08678ff785df"} Oct 29 23:07:24.313195 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:24.313118 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" podStartSLOduration=1.313099357 podStartE2EDuration="1.313099357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:24.307312973 +0000 UTC m=+1498.247860888" watchObservedRunningTime="2026-10-29 23:07:24.313099357 +0000 UTC m=+1498.253647268" Oct 29 23:07:24.416336 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:24.413473 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:24.432030 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:24.431995 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 29 23:07:24.432235 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:24.432053 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" podUID="041cb60f-5e7c-47d7-8e83-cce35d4c9529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 29 23:07:25.274155 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:25.274065 4821 generic.go:356] "Generic (PLEG): container finished" podID="ff6b1260-3dbc-4391-a1a9-7c2de989026c" containerID="59b5413d902abc9442d1e9976f6e4c306a237d1d9b34c974b04adf31e18bbb64" exitCode=0 Oct 29 23:07:25.274155 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:25.274135 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" event={"ID":"ff6b1260-3dbc-4391-a1a9-7c2de989026c","Type":"ContainerDied","Data":"59b5413d902abc9442d1e9976f6e4c306a237d1d9b34c974b04adf31e18bbb64"} Oct 29 23:07:25.274802 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:25.274623 4821 scope.go:122] "RemoveContainer" containerID="59b5413d902abc9442d1e9976f6e4c306a237d1d9b34c974b04adf31e18bbb64" Oct 29 23:07:25.276670 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:25.276640 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:07:25.276786 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:25.276726 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:07:25.276786 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:25.276775 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:07:25.412330 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:25.412303 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 29 23:07:25.412471 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:25.412355 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" podUID="041cb60f-5e7c-47d7-8e83-cce35d4c9529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 29 23:07:25.645285 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:25.639540 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30791c5-jjf5n" Oct 29 23:07:25.774746 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:25.774706 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb36af1-9a9c-45fc-9996-ffa8bfd7d690-kube-api-access-q69jm\" (UniqueName: \"kubernetes.io/projected/1fb36af1-9a9c-45fc-9996-ffa8bfd7d690-kube-api-access-q69jm\") pod \"1fb36af1-9a9c-45fc-9996-ffa8bfd7d690\" (UID: \"1fb36af1-9a9c-45fc-9996-ffa8bfd7d690\") " Oct 29 23:07:25.774948 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:25.774773 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fb36af1-9a9c-45fc-9996-ffa8bfd7d690-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fb36af1-9a9c-45fc-9996-ffa8bfd7d690-maas-api-xks-ca\") pod \"1fb36af1-9a9c-45fc-9996-ffa8bfd7d690\" (UID: \"1fb36af1-9a9c-45fc-9996-ffa8bfd7d690\") " Oct 29 23:07:25.774948 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:25.774900 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fb36af1-9a9c-45fc-9996-ffa8bfd7d690-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fb36af1-9a9c-45fc-9996-ffa8bfd7d690-maas-api-openshift-ca\") pod \"1fb36af1-9a9c-45fc-9996-ffa8bfd7d690\" (UID: \"1fb36af1-9a9c-45fc-9996-ffa8bfd7d690\") " Oct 29 23:07:25.775605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:25.775578 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fb36af1-9a9c-45fc-9996-ffa8bfd7d690-maas-api-openshift-ca" pod "1fb36af1-9a9c-45fc-9996-ffa8bfd7d690" (UID: "1fb36af1-9a9c-45fc-9996-ffa8bfd7d690"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:25.777221 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:25.777111 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fb36af1-9a9c-45fc-9996-ffa8bfd7d690-maas-api-xks-ca" pod "1fb36af1-9a9c-45fc-9996-ffa8bfd7d690" (UID: "1fb36af1-9a9c-45fc-9996-ffa8bfd7d690"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:25.777221 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:25.777145 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb36af1-9a9c-45fc-9996-ffa8bfd7d690-kube-api-access-q69jm" pod "1fb36af1-9a9c-45fc-9996-ffa8bfd7d690" (UID: "1fb36af1-9a9c-45fc-9996-ffa8bfd7d690"). InnerVolumeSpecName "kube-api-access-q69jm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:25.877333 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:25.877297 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fb36af1-9a9c-45fc-9996-ffa8bfd7d690-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:25.877333 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:25.877336 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fb36af1-9a9c-45fc-9996-ffa8bfd7d690-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:25.877541 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:25.877352 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q69jm\" (UniqueName: \"kubernetes.io/projected/1fb36af1-9a9c-45fc-9996-ffa8bfd7d690-kube-api-access-q69jm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:26.288732 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.288699 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" event={"ID":"ff6b1260-3dbc-4391-a1a9-7c2de989026c","Type":"ContainerStarted","Data":"408ae427853f06dc48f45ead4edaba3d9ce7ec4c60a3e9dd6cf15166125b1b16"} Oct 29 23:07:26.290165 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.290149 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30791c5-jjf5n" Oct 29 23:07:26.290244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.290188 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b30791c5-jjf5n" event={"ID":"1fb36af1-9a9c-45fc-9996-ffa8bfd7d690","Type":"ContainerDied","Data":"6cd7ebbfcc38dfed1c32b343466ec44a4b654c1ac54cf5963bbe02f0c3b9b6c1"} Oct 29 23:07:26.290285 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.290266 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cd7ebbfcc38dfed1c32b343466ec44a4b654c1ac54cf5963bbe02f0c3b9b6c1" Oct 29 23:07:26.412864 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.412519 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 29 23:07:26.412864 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.412576 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" podUID="041cb60f-5e7c-47d7-8e83-cce35d4c9529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 29 23:07:26.672670 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.672638 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz"] Oct 29 23:07:26.673350 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.673334 4821 cpu_manager.go:414] "containerMap: removing container" podUID="1fb36af1-9a9c-45fc-9996-ffa8bfd7d690" containerName="revoke-keys" Oct 29 23:07:26.673412 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.673351 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb36af1-9a9c-45fc-9996-ffa8bfd7d690" containerName="revoke-keys" Oct 29 23:07:26.673462 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.673450 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb36af1-9a9c-45fc-9996-ffa8bfd7d690" containerName="revoke-keys" Oct 29 23:07:26.677361 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.677130 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.690478 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.690448 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q5c6\" (UniqueName: \"kubernetes.io/projected/3adc54c5-a6cf-413f-a347-1397731cf3ff-kube-api-access-5q5c6\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.690579 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.690512 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-workload-certs\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.690579 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.690552 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-credential-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.690699 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.690622 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-workload-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.690699 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.690679 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-envoy\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.690903 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.690885 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-token\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.690959 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.690920 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-data\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.691035 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.691018 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3adc54c5-a6cf-413f-a347-1397731cf3ff-istiod-ca-cert\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.691175 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.691093 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-podinfo\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.698287 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.698257 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk"] Oct 29 23:07:26.703719 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.703698 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.703959 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.703935 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz"] Oct 29 23:07:26.757513 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.740479 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:07:26.763246 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.748351 4821 patch_prober.go:29] interesting pod/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 29 23:07:26.783328 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.783289 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk"] Oct 29 23:07:26.783576 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.783311 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" podUID="ff6b1260-3dbc-4391-a1a9-7c2de989026c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 29 23:07:26.793039 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.792925 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-workload-certs\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.793039 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.792967 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-data\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.793272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.793132 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q5c6\" (UniqueName: \"kubernetes.io/projected/3adc54c5-a6cf-413f-a347-1397731cf3ff-kube-api-access-5q5c6\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.793272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.793166 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istiod-ca-cert\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.793272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.793217 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-workload-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.793272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.793253 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-workload-certs\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.793487 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.793293 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-credential-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.793487 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.793362 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-workload-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.793487 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.793422 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-envoy\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.793487 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.793451 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-credential-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.793487 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.793480 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-envoy\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.793740 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.793516 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-token\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.793740 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.793610 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-podinfo\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.793740 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.793648 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-token\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.793740 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.793674 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-data\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.793946 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.793759 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcctb\" (UniqueName: \"kubernetes.io/projected/b6671cb5-2dc2-4ced-9aee-33c298a0659e-kube-api-access-gcctb\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.793946 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.793813 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3adc54c5-a6cf-413f-a347-1397731cf3ff-istiod-ca-cert\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.793946 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.793851 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-podinfo\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.794100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.794079 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-credential-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.794579 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.794551 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-workload-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.794790 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.794763 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-workload-certs\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.795017 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.795003 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3adc54c5-a6cf-413f-a347-1397731cf3ff-istiod-ca-cert\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.795155 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.795130 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-data\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.801234 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.800744 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-podinfo\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.801234 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.800928 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-envoy\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.881688 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.880761 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-token\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.884676 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.884638 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q5c6\" (UniqueName: \"kubernetes.io/projected/3adc54c5-a6cf-413f-a347-1397731cf3ff-kube-api-access-5q5c6\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.897225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.896288 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-credential-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.897225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.896332 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-envoy\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.897225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.896366 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-token\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.897225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.896460 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-podinfo\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.897225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.896562 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcctb\" (UniqueName: \"kubernetes.io/projected/b6671cb5-2dc2-4ced-9aee-33c298a0659e-kube-api-access-gcctb\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.897225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.896669 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-workload-certs\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.897225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.896730 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-data\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.897225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.896845 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istiod-ca-cert\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.897225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.896971 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-workload-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.900765 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.898043 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-workload-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.900765 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.898585 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-data\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.903603 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.903189 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istiod-ca-cert\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.906545 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.906509 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-envoy\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.912897 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.909119 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-credential-socket\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.912897 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.909480 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-workload-certs\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.996719 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.996622 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcctb\" (UniqueName: \"kubernetes.io/projected/b6671cb5-2dc2-4ced-9aee-33c298a0659e-kube-api-access-gcctb\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.998392 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.998331 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-token\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:26.998853 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.998543 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:26.998853 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:26.998781 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-podinfo\") pod \"e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:27.046680 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:27.046645 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:27.412281 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:27.412252 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 29 23:07:27.412671 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:27.412301 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" podUID="041cb60f-5e7c-47d7-8e83-cce35d4c9529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 29 23:07:27.486239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:27.484770 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz"] Oct 29 23:07:27.486239 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:07:27.485419 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6671cb5_2dc2_4ced_9aee_33c298a0659e.slice/crio-e0d7d3bd41353768e33b968e89f2f7b4efb29fb626d314c803b24bb1f020a862 WatchSource:0}: Error finding container e0d7d3bd41353768e33b968e89f2f7b4efb29fb626d314c803b24bb1f020a862: Status 404 returned error can't find the container with id e0d7d3bd41353768e33b968e89f2f7b4efb29fb626d314c803b24bb1f020a862 Oct 29 23:07:27.487398 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:27.487365 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:07:27.487519 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:27.487432 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:07:27.487519 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:27.487461 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:07:27.490141 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:27.490110 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:07:27.490295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:27.490220 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:07:27.490295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:27.490280 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:07:27.498021 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:27.497998 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk"] Oct 29 23:07:27.736229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:27.735804 4821 patch_prober.go:29] interesting pod/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 29 23:07:27.736229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:27.735866 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" podUID="ff6b1260-3dbc-4391-a1a9-7c2de989026c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 29 23:07:28.338229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:28.319890 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" event={"ID":"3adc54c5-a6cf-413f-a347-1397731cf3ff","Type":"ContainerStarted","Data":"6dbeb918ccc35bc6ae20ac6ff4cad4e2384b04db70a95d70ab4ae51838511fa3"} Oct 29 23:07:28.338229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:28.319956 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" event={"ID":"3adc54c5-a6cf-413f-a347-1397731cf3ff","Type":"ContainerStarted","Data":"cb3c4e7f8fde960ee7b9db156fb77ce115688797388384f27868248b560e658e"} Oct 29 23:07:28.359915 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:28.345232 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" event={"ID":"b6671cb5-2dc2-4ced-9aee-33c298a0659e","Type":"ContainerStarted","Data":"d458af9722a3e4b31d2b7e84871ff6736e0a3fe0af705aaf5fde4beda8dae015"} Oct 29 23:07:28.359915 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:28.345315 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" event={"ID":"b6671cb5-2dc2-4ced-9aee-33c298a0659e","Type":"ContainerStarted","Data":"e0d7d3bd41353768e33b968e89f2f7b4efb29fb626d314c803b24bb1f020a862"} Oct 29 23:07:28.409424 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:28.409375 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" podStartSLOduration=2.409361015 podStartE2EDuration="2.409361015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:28.390665712 +0000 UTC m=+1502.331213624" watchObservedRunningTime="2026-10-29 23:07:28.409361015 +0000 UTC m=+1502.349908924" Oct 29 23:07:28.411879 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:28.411855 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 29 23:07:28.412052 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:28.412035 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" podUID="041cb60f-5e7c-47d7-8e83-cce35d4c9529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 29 23:07:28.464439 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:28.463229 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" podStartSLOduration=2.463188741 podStartE2EDuration="2.463188741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:28.459773139 +0000 UTC m=+1502.400321062" watchObservedRunningTime="2026-10-29 23:07:28.463188741 +0000 UTC m=+1502.403736652" Oct 29 23:07:28.730675 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:28.730633 4821 patch_prober.go:29] interesting pod/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 29 23:07:28.730962 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:28.730932 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" podUID="ff6b1260-3dbc-4391-a1a9-7c2de989026c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 29 23:07:29.008140 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:29.008048 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:29.030533 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:29.030496 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 29 23:07:29.030813 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:29.030784 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" podUID="3adc54c5-a6cf-413f-a347-1397731cf3ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 29 23:07:29.083538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:29.066941 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:29.105229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:29.084731 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 29 23:07:29.105229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:29.084791 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" podUID="b6671cb5-2dc2-4ced-9aee-33c298a0659e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 29 23:07:29.209226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:29.208690 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:07:29.414245 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:29.413181 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 29 23:07:29.414245 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:29.413264 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" podUID="041cb60f-5e7c-47d7-8e83-cce35d4c9529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 29 23:07:29.478227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:29.473724 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp"] Oct 29 23:07:29.787280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:29.742400 4821 patch_prober.go:29] interesting pod/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 29 23:07:29.787280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:29.742476 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" podUID="ff6b1260-3dbc-4391-a1a9-7c2de989026c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 29 23:07:29.952257 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:29.951492 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:29.996261 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:29.952581 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:30.076282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.030352 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 29 23:07:30.076282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.030418 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" podUID="3adc54c5-a6cf-413f-a347-1397731cf3ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 29 23:07:30.076282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.064358 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 29 23:07:30.076282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.064419 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" podUID="b6671cb5-2dc2-4ced-9aee-33c298a0659e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 29 23:07:30.221228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.220447 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:30.237450 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.236097 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:30.264787 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.258564 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded" Oct 29 23:07:30.403339 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.403294 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 29 23:07:30.403963 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.403933 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 29 23:07:30.409040 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.405288 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:30.416093 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.416013 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:30.418687 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.418580 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 29 23:07:30.418687 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.418641 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" podUID="041cb60f-5e7c-47d7-8e83-cce35d4c9529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 29 23:07:30.468610 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.468565 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:30.566751 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.564283 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:30.629611 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.629566 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:30.632304 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.632271 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 29 23:07:30.731373 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.731190 4821 patch_prober.go:29] interesting pod/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 29 23:07:30.731696 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.731665 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" podUID="ff6b1260-3dbc-4391-a1a9-7c2de989026c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 29 23:07:30.744352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.744289 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 29 23:07:30.746659 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.746629 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:30.762400 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.762367 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:30.805390 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.805344 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" podUID="0a560010-b749-4e0d-875c-4b5d880410ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.155:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:30.964231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.960774 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:30.964231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.961177 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded" Oct 29 23:07:30.964231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:30.961630 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 23:07:31.000293 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.000148 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 29 23:07:31.000293 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.000239 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" podUID="3adc54c5-a6cf-413f-a347-1397731cf3ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 29 23:07:31.114709 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.114674 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 29 23:07:31.131604 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.114967 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" podUID="b6671cb5-2dc2-4ced-9aee-33c298a0659e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 29 23:07:31.241095 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.238518 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:31.241095 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.239157 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 23:07:31.402290 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.398942 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 23:07:31.406300 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.406263 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 29 23:07:31.407025 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.407003 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 23:07:31.409369 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.409286 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:31.410072 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.410055 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 23:07:31.412104 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.412084 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 29 23:07:31.581229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.422470 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 29 23:07:31.581229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.496558 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 29 23:07:31.581229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.503856 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded" Oct 29 23:07:31.581229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.555293 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" podUID="0a560010-b749-4e0d-875c-4b5d880410ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.155:8000/ready\": context deadline exceeded" Oct 29 23:07:31.581229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.571344 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" podUID="041cb60f-5e7c-47d7-8e83-cce35d4c9529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 29 23:07:31.581229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.579433 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 23:07:31.581229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.581090 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 23:07:31.597229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.591347 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:31.597229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.591813 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 23:07:31.631559 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.631518 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:31.632291 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.632262 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 23:07:31.689331 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.687294 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 29 23:07:31.718228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.705351 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 23:07:31.809228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.742453 4821 patch_prober.go:29] interesting pod/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 29 23:07:31.809228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.742510 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" podUID="ff6b1260-3dbc-4391-a1a9-7c2de989026c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 29 23:07:31.817896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.817843 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:31.818638 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.818616 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 23:07:31.870373 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.845473 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 29 23:07:31.878229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:31.870725 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 23:07:32.029282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:32.003574 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 29 23:07:32.125316 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:32.125273 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" podUID="3adc54c5-a6cf-413f-a347-1397731cf3ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 29 23:07:32.125571 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:32.115971 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 29 23:07:32.129224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:32.125673 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" podUID="b6671cb5-2dc2-4ced-9aee-33c298a0659e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 29 23:07:32.445563 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:32.420108 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 23:07:32.445563 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:32.439367 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:32.473257 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:32.449319 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 29 23:07:32.473257 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:32.449648 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:32.473257 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:32.462475 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 29 23:07:32.473257 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:32.462538 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" podUID="041cb60f-5e7c-47d7-8e83-cce35d4c9529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 29 23:07:32.473768 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:32.473737 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:32.656487 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:32.656429 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:32.657243 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:32.657182 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" podUID="0a560010-b749-4e0d-875c-4b5d880410ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.155:8000/ready\": context deadline exceeded" Oct 29 23:07:32.662327 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:32.657239 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:32.662327 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:32.657312 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 23:07:32.662761 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:32.658513 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:32.696486 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:32.696368 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:32.710422 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:32.710373 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:07:32.762975 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:32.762943 4821 patch_prober.go:29] interesting pod/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 29 23:07:32.763329 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:32.763298 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" podUID="ff6b1260-3dbc-4391-a1a9-7c2de989026c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 29 23:07:32.878351 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:32.858505 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:32.878351 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:32.859871 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:33.061673 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:33.061474 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 29 23:07:33.062122 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:33.061999 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" podUID="b6671cb5-2dc2-4ced-9aee-33c298a0659e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 29 23:07:33.091012 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:33.090979 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 29 23:07:33.114915 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:33.091292 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" podUID="3adc54c5-a6cf-413f-a347-1397731cf3ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 29 23:07:33.261371 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:33.261320 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:33.452568 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:33.452517 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:33.462603 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:33.462557 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:33.464074 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:33.464038 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:33.470505 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:33.470285 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:33.475628 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:33.475606 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 23:07:33.545331 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:33.545288 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:33.546913 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:33.546888 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 23:07:33.607227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:33.597001 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 23:07:33.725286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:33.672274 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:33.725286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:33.679271 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:33.725286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:33.697550 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:33.725286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:33.717051 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" podUID="0a560010-b749-4e0d-875c-4b5d880410ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.155:8000/ready\": context deadline exceeded" Oct 29 23:07:33.725286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:33.721293 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 29 23:07:33.725286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:33.723392 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:33.813045 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:33.813002 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:33.842259 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:33.816503 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:07:33.845183 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:33.845147 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:07:33.861527 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:33.860301 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 29 23:07:34.001533 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:34.001432 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:07:34.012746 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:34.012714 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 23:07:34.041472 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:34.041012 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 29 23:07:34.041472 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:34.041070 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" podUID="3adc54c5-a6cf-413f-a347-1397731cf3ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 29 23:07:34.081231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:34.067327 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:34.096389 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:34.095991 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 29 23:07:34.096389 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:34.096047 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" podUID="b6671cb5-2dc2-4ced-9aee-33c298a0659e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 29 23:07:34.100088 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:34.098523 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 23:07:34.206944 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:34.206908 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 23:07:34.207280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:34.207260 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 23:07:34.207507 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:34.207485 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 23:07:34.295649 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:34.294607 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:34.296527 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:34.296408 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 23:07:34.302091 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:34.301932 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 23:07:34.303431 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:34.303277 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 23:07:34.303620 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:34.303601 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 23:07:34.305665 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:34.304270 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 23:07:34.308581 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:34.308558 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 23:07:34.395872 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:34.395845 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:34.397324 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:34.397304 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:34.399589 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:34.399571 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:07:34.410451 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:34.410421 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 23:07:34.419230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:34.412658 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 23:07:34.489791 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:34.475295 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp" podUID="7c03888c-f9e6-4f33-95bb-e17e22d4ff54" containerName="payload-pre-processing" containerID="cri-o://0fdcbaa06d609e99160aeed362815f54fddec044d1539a5ab897b200bc301f74" gracePeriod=25 Oct 29 23:07:35.000886 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:35.000847 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 29 23:07:35.001495 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:35.000907 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" podUID="3adc54c5-a6cf-413f-a347-1397731cf3ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 29 23:07:35.052284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:35.050347 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 29 23:07:35.052284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:35.050432 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" podUID="b6671cb5-2dc2-4ced-9aee-33c298a0659e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 29 23:07:35.372237 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:35.365633 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp" Oct 29 23:07:35.408586 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:35.408555 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c03888c-f9e6-4f33-95bb-e17e22d4ff54-kube-api-access-k7v7d\" (UniqueName: \"kubernetes.io/projected/7c03888c-f9e6-4f33-95bb-e17e22d4ff54-kube-api-access-k7v7d\") pod \"7c03888c-f9e6-4f33-95bb-e17e22d4ff54\" (UID: \"7c03888c-f9e6-4f33-95bb-e17e22d4ff54\") " Oct 29 23:07:35.408811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:35.408779 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c03888c-f9e6-4f33-95bb-e17e22d4ff54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c03888c-f9e6-4f33-95bb-e17e22d4ff54-plugins-config-volume\") pod \"7c03888c-f9e6-4f33-95bb-e17e22d4ff54\" (UID: \"7c03888c-f9e6-4f33-95bb-e17e22d4ff54\") " Oct 29 23:07:35.409109 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:35.409086 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c03888c-f9e6-4f33-95bb-e17e22d4ff54-plugins-config-volume" pod "7c03888c-f9e6-4f33-95bb-e17e22d4ff54" (UID: "7c03888c-f9e6-4f33-95bb-e17e22d4ff54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:35.409999 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:35.409972 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c03888c-f9e6-4f33-95bb-e17e22d4ff54-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:35.474218 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:35.474141 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c03888c-f9e6-4f33-95bb-e17e22d4ff54-kube-api-access-k7v7d" pod "7c03888c-f9e6-4f33-95bb-e17e22d4ff54" (UID: "7c03888c-f9e6-4f33-95bb-e17e22d4ff54"). InnerVolumeSpecName "kube-api-access-k7v7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:35.512367 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:35.512331 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7v7d\" (UniqueName: \"kubernetes.io/projected/7c03888c-f9e6-4f33-95bb-e17e22d4ff54-kube-api-access-k7v7d\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:35.654189 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:35.654147 4821 generic.go:356] "Generic (PLEG): container finished" podID="7c03888c-f9e6-4f33-95bb-e17e22d4ff54" containerID="0fdcbaa06d609e99160aeed362815f54fddec044d1539a5ab897b200bc301f74" exitCode=0 Oct 29 23:07:35.655376 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:35.655278 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp" Oct 29 23:07:35.655704 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:35.655676 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp" event={"ID":"7c03888c-f9e6-4f33-95bb-e17e22d4ff54","Type":"ContainerDied","Data":"0fdcbaa06d609e99160aeed362815f54fddec044d1539a5ab897b200bc301f74"} Oct 29 23:07:35.655943 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:35.655752 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp" event={"ID":"7c03888c-f9e6-4f33-95bb-e17e22d4ff54","Type":"ContainerDied","Data":"df789ee262939d3f33b490cbf144e7381c30198e03e6d8925c3e66861266db62"} Oct 29 23:07:35.655943 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:35.655781 4821 scope.go:122] "RemoveContainer" containerID="0fdcbaa06d609e99160aeed362815f54fddec044d1539a5ab897b200bc301f74" Oct 29 23:07:35.695121 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:35.695095 4821 scope.go:122] "RemoveContainer" containerID="0fdcbaa06d609e99160aeed362815f54fddec044d1539a5ab897b200bc301f74" Oct 29 23:07:35.696619 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:07:35.696573 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fdcbaa06d609e99160aeed362815f54fddec044d1539a5ab897b200bc301f74\": container with ID starting with 0fdcbaa06d609e99160aeed362815f54fddec044d1539a5ab897b200bc301f74 not found: ID does not exist" containerID="0fdcbaa06d609e99160aeed362815f54fddec044d1539a5ab897b200bc301f74" Oct 29 23:07:35.696730 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:35.696634 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fdcbaa06d609e99160aeed362815f54fddec044d1539a5ab897b200bc301f74"} err="failed to get container status \"0fdcbaa06d609e99160aeed362815f54fddec044d1539a5ab897b200bc301f74\": rpc error: code = NotFound desc = could not find container \"0fdcbaa06d609e99160aeed362815f54fddec044d1539a5ab897b200bc301f74\": container with ID starting with 0fdcbaa06d609e99160aeed362815f54fddec044d1539a5ab897b200bc301f74 not found: ID does not exist" Oct 29 23:07:35.727287 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:35.727190 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp"] Oct 29 23:07:35.732490 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:35.732458 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3f7d4b-79d47b947f-48prp"] Oct 29 23:07:36.007731 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:36.007650 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:36.008338 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:36.008275 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:36.010219 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:36.010183 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:07:36.052869 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:36.052837 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:36.053256 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:36.053237 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:36.053940 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:36.053918 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:07:36.174849 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:36.174815 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c03888c-f9e6-4f33-95bb-e17e22d4ff54" path="/var/lib/kubelet/pods/7c03888c-f9e6-4f33-95bb-e17e22d4ff54/volumes" Oct 29 23:07:49.253468 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:49.253435 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c"] Oct 29 23:07:49.267304 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:49.267263 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d67deb-z465c"] Oct 29 23:07:50.172061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:50.172026 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6a4efc6-779b-4265-8aa4-00b7dbaa15b0" path="/var/lib/kubelet/pods/b6a4efc6-779b-4265-8aa4-00b7dbaa15b0/volumes" Oct 29 23:07:51.467401 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:51.467371 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd"] Oct 29 23:07:51.467854 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:51.467637 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" podUID="d648624c-86e0-4640-8e90-048cd2c02d19" containerName="maas-api" containerID="cri-o://c24b43949b00e6245399d91accc86bd3299745c124aa5d06c51afe74a9df32f5" gracePeriod=30 Oct 29 23:07:51.841897 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:51.841867 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" Oct 29 23:07:51.852923 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:51.852885 4821 generic.go:356] "Generic (PLEG): container finished" podID="d648624c-86e0-4640-8e90-048cd2c02d19" containerID="c24b43949b00e6245399d91accc86bd3299745c124aa5d06c51afe74a9df32f5" exitCode=0 Oct 29 23:07:51.853108 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:51.852928 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" event={"ID":"d648624c-86e0-4640-8e90-048cd2c02d19","Type":"ContainerDied","Data":"c24b43949b00e6245399d91accc86bd3299745c124aa5d06c51afe74a9df32f5"} Oct 29 23:07:51.853108 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:51.852954 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" Oct 29 23:07:51.853108 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:51.852999 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd" event={"ID":"d648624c-86e0-4640-8e90-048cd2c02d19","Type":"ContainerDied","Data":"62fcd5289d58be748bbf1a639a986798deae51ad067b57d66744e03af2a255f7"} Oct 29 23:07:51.853108 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:51.853034 4821 scope.go:122] "RemoveContainer" containerID="c24b43949b00e6245399d91accc86bd3299745c124aa5d06c51afe74a9df32f5" Oct 29 23:07:51.865075 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:51.865057 4821 scope.go:122] "RemoveContainer" containerID="c24b43949b00e6245399d91accc86bd3299745c124aa5d06c51afe74a9df32f5" Oct 29 23:07:51.865391 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:07:51.865369 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c24b43949b00e6245399d91accc86bd3299745c124aa5d06c51afe74a9df32f5\": container with ID starting with c24b43949b00e6245399d91accc86bd3299745c124aa5d06c51afe74a9df32f5 not found: ID does not exist" containerID="c24b43949b00e6245399d91accc86bd3299745c124aa5d06c51afe74a9df32f5" Oct 29 23:07:51.865467 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:51.865403 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c24b43949b00e6245399d91accc86bd3299745c124aa5d06c51afe74a9df32f5"} err="failed to get container status \"c24b43949b00e6245399d91accc86bd3299745c124aa5d06c51afe74a9df32f5\": rpc error: code = NotFound desc = could not find container \"c24b43949b00e6245399d91accc86bd3299745c124aa5d06c51afe74a9df32f5\": container with ID starting with c24b43949b00e6245399d91accc86bd3299745c124aa5d06c51afe74a9df32f5 not found: ID does not exist" Oct 29 23:07:51.889863 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:51.889833 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d648624c-86e0-4640-8e90-048cd2c02d19-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d648624c-86e0-4640-8e90-048cd2c02d19-metrics-certs\") pod \"d648624c-86e0-4640-8e90-048cd2c02d19\" (UID: \"d648624c-86e0-4640-8e90-048cd2c02d19\") " Oct 29 23:07:51.890120 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:51.890098 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d648624c-86e0-4640-8e90-048cd2c02d19-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d648624c-86e0-4640-8e90-048cd2c02d19-maas-api-tls\") pod \"d648624c-86e0-4640-8e90-048cd2c02d19\" (UID: \"d648624c-86e0-4640-8e90-048cd2c02d19\") " Oct 29 23:07:51.890406 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:51.890386 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d648624c-86e0-4640-8e90-048cd2c02d19-kube-api-access-cqg2c\" (UniqueName: \"kubernetes.io/projected/d648624c-86e0-4640-8e90-048cd2c02d19-kube-api-access-cqg2c\") pod \"d648624c-86e0-4640-8e90-048cd2c02d19\" (UID: \"d648624c-86e0-4640-8e90-048cd2c02d19\") " Oct 29 23:07:51.890703 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:51.890677 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d648624c-86e0-4640-8e90-048cd2c02d19-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d648624c-86e0-4640-8e90-048cd2c02d19-openshift-service-ca\") pod \"d648624c-86e0-4640-8e90-048cd2c02d19\" (UID: \"d648624c-86e0-4640-8e90-048cd2c02d19\") " Oct 29 23:07:51.891630 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:51.891604 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d648624c-86e0-4640-8e90-048cd2c02d19-openshift-service-ca" pod "d648624c-86e0-4640-8e90-048cd2c02d19" (UID: "d648624c-86e0-4640-8e90-048cd2c02d19"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:51.893065 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:51.893034 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d648624c-86e0-4640-8e90-048cd2c02d19-maas-api-tls" pod "d648624c-86e0-4640-8e90-048cd2c02d19" (UID: "d648624c-86e0-4640-8e90-048cd2c02d19"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:51.894505 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:51.894351 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d648624c-86e0-4640-8e90-048cd2c02d19-kube-api-access-cqg2c" pod "d648624c-86e0-4640-8e90-048cd2c02d19" (UID: "d648624c-86e0-4640-8e90-048cd2c02d19"). InnerVolumeSpecName "kube-api-access-cqg2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:51.900634 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:51.900605 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d648624c-86e0-4640-8e90-048cd2c02d19-metrics-certs" pod "d648624c-86e0-4640-8e90-048cd2c02d19" (UID: "d648624c-86e0-4640-8e90-048cd2c02d19"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:51.993067 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:51.993028 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d648624c-86e0-4640-8e90-048cd2c02d19-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:51.993067 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:51.993071 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d648624c-86e0-4640-8e90-048cd2c02d19-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:51.993354 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:51.993085 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d648624c-86e0-4640-8e90-048cd2c02d19-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:51.993354 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:51.993098 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqg2c\" (UniqueName: \"kubernetes.io/projected/d648624c-86e0-4640-8e90-048cd2c02d19-kube-api-access-cqg2c\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:52.252282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:52.252159 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd"] Oct 29 23:07:52.259955 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:52.258985 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7950-58cdc5c4fd-6n2bd"] Oct 29 23:07:53.888522 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:07:53.888481 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:07:53.945775 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:53.945740 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl"] Oct 29 23:07:53.946104 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:53.946049 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl" podUID="8b0089e6-032c-4d01-8557-987929bb2de7" containerName="maas-api" containerID="cri-o://3630394ae606a60d7ac713056d032a7cdf75e4f7ae0bf104fd19f23ec8812c2c" gracePeriod=30 Oct 29 23:07:54.475261 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.459408 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d648624c-86e0-4640-8e90-048cd2c02d19" path="/var/lib/kubelet/pods/d648624c-86e0-4640-8e90-048cd2c02d19/volumes" Oct 29 23:07:54.527234 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.522688 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl" Oct 29 23:07:54.623531 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.623499 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b0089e6-032c-4d01-8557-987929bb2de7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b0089e6-032c-4d01-8557-987929bb2de7-openshift-service-ca\") pod \"8b0089e6-032c-4d01-8557-987929bb2de7\" (UID: \"8b0089e6-032c-4d01-8557-987929bb2de7\") " Oct 29 23:07:54.641720 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.641177 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b0089e6-032c-4d01-8557-987929bb2de7-openshift-service-ca" pod "8b0089e6-032c-4d01-8557-987929bb2de7" (UID: "8b0089e6-032c-4d01-8557-987929bb2de7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:54.644754 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.644725 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b0089e6-032c-4d01-8557-987929bb2de7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b0089e6-032c-4d01-8557-987929bb2de7-maas-api-tls\") pod \"8b0089e6-032c-4d01-8557-987929bb2de7\" (UID: \"8b0089e6-032c-4d01-8557-987929bb2de7\") " Oct 29 23:07:54.645226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.645183 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b0089e6-032c-4d01-8557-987929bb2de7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b0089e6-032c-4d01-8557-987929bb2de7-metrics-certs\") pod \"8b0089e6-032c-4d01-8557-987929bb2de7\" (UID: \"8b0089e6-032c-4d01-8557-987929bb2de7\") " Oct 29 23:07:54.646475 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.646440 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b0089e6-032c-4d01-8557-987929bb2de7-kube-api-access-r7gwk\" (UniqueName: \"kubernetes.io/projected/8b0089e6-032c-4d01-8557-987929bb2de7-kube-api-access-r7gwk\") pod \"8b0089e6-032c-4d01-8557-987929bb2de7\" (UID: \"8b0089e6-032c-4d01-8557-987929bb2de7\") " Oct 29 23:07:54.647785 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.647763 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b0089e6-032c-4d01-8557-987929bb2de7-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:54.648122 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.648023 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b0089e6-032c-4d01-8557-987929bb2de7-metrics-certs" pod "8b0089e6-032c-4d01-8557-987929bb2de7" (UID: "8b0089e6-032c-4d01-8557-987929bb2de7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:54.650923 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.650386 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b0089e6-032c-4d01-8557-987929bb2de7-maas-api-tls" pod "8b0089e6-032c-4d01-8557-987929bb2de7" (UID: "8b0089e6-032c-4d01-8557-987929bb2de7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:54.652427 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.652401 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b0089e6-032c-4d01-8557-987929bb2de7-kube-api-access-r7gwk" pod "8b0089e6-032c-4d01-8557-987929bb2de7" (UID: "8b0089e6-032c-4d01-8557-987929bb2de7"). InnerVolumeSpecName "kube-api-access-r7gwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:54.739638 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.739471 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4"] Oct 29 23:07:54.749352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.749321 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b0089e6-032c-4d01-8557-987929bb2de7-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:54.749352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.749346 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b0089e6-032c-4d01-8557-987929bb2de7-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:54.749352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.749356 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7gwk\" (UniqueName: \"kubernetes.io/projected/8b0089e6-032c-4d01-8557-987929bb2de7-kube-api-access-r7gwk\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:07:54.882319 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.882287 4821 generic.go:356] "Generic (PLEG): container finished" podID="8b0089e6-032c-4d01-8557-987929bb2de7" containerID="3630394ae606a60d7ac713056d032a7cdf75e4f7ae0bf104fd19f23ec8812c2c" exitCode=0 Oct 29 23:07:54.882521 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.882334 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl" Oct 29 23:07:54.882521 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.882372 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl" event={"ID":"8b0089e6-032c-4d01-8557-987929bb2de7","Type":"ContainerDied","Data":"3630394ae606a60d7ac713056d032a7cdf75e4f7ae0bf104fd19f23ec8812c2c"} Oct 29 23:07:54.882521 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.882439 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl" event={"ID":"8b0089e6-032c-4d01-8557-987929bb2de7","Type":"ContainerDied","Data":"25e1b58bc2a038e763ee4c5ab58eac3f52fa2e52f791ab462e05e809f1cd3973"} Oct 29 23:07:54.882521 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.882470 4821 scope.go:122] "RemoveContainer" containerID="3630394ae606a60d7ac713056d032a7cdf75e4f7ae0bf104fd19f23ec8812c2c" Oct 29 23:07:54.893950 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.893928 4821 scope.go:122] "RemoveContainer" containerID="3630394ae606a60d7ac713056d032a7cdf75e4f7ae0bf104fd19f23ec8812c2c" Oct 29 23:07:54.894281 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:07:54.894263 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3630394ae606a60d7ac713056d032a7cdf75e4f7ae0bf104fd19f23ec8812c2c\": container with ID starting with 3630394ae606a60d7ac713056d032a7cdf75e4f7ae0bf104fd19f23ec8812c2c not found: ID does not exist" containerID="3630394ae606a60d7ac713056d032a7cdf75e4f7ae0bf104fd19f23ec8812c2c" Oct 29 23:07:54.894334 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.894290 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3630394ae606a60d7ac713056d032a7cdf75e4f7ae0bf104fd19f23ec8812c2c"} err="failed to get container status \"3630394ae606a60d7ac713056d032a7cdf75e4f7ae0bf104fd19f23ec8812c2c\": rpc error: code = NotFound desc = could not find container \"3630394ae606a60d7ac713056d032a7cdf75e4f7ae0bf104fd19f23ec8812c2c\": container with ID starting with 3630394ae606a60d7ac713056d032a7cdf75e4f7ae0bf104fd19f23ec8812c2c not found: ID does not exist" Oct 29 23:07:54.941155 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.941118 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl"] Oct 29 23:07:54.947871 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.947833 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f7d4b-f8db8c6f8-shrvl"] Oct 29 23:07:54.964570 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.964527 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd"] Oct 29 23:07:54.972491 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:54.972424 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3f7d4b-p6wkd"] Oct 29 23:07:56.171618 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:56.171587 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b0089e6-032c-4d01-8557-987929bb2de7" path="/var/lib/kubelet/pods/8b0089e6-032c-4d01-8557-987929bb2de7/volumes" Oct 29 23:07:56.172296 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:07:56.172274 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be298eab-1385-494b-b0a1-97930eede6bd" path="/var/lib/kubelet/pods/be298eab-1385-494b-b0a1-97930eede6bd/volumes" Oct 29 23:08:00.158897 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.158860 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl"] Oct 29 23:08:00.159393 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.159134 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" podUID="914c88e0-ed28-42c6-b01d-1be633f49e4b" containerName="istio-proxy" containerID="cri-o://c7790336fb3c251d1bc383f9d9109d8673895974bceb5234bc408391ce74a67c" gracePeriod=30 Oct 29 23:08:00.174488 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.174463 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd"] Oct 29 23:08:00.174710 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.174675 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" podUID="f1840a05-a8fe-40e2-b7f9-39a91f6f355f" containerName="istio-proxy" containerID="cri-o://7e9cb7a0b247a2a1ce0018539a2a1ef4ba002451743af20747de0a6fa7c5c5d6" gracePeriod=30 Oct 29 23:08:00.738389 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.738353 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq"] Oct 29 23:08:00.738937 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.738924 4821 cpu_manager.go:414] "containerMap: removing container" podUID="8b0089e6-032c-4d01-8557-987929bb2de7" containerName="maas-api" Oct 29 23:08:00.738983 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.738939 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b0089e6-032c-4d01-8557-987929bb2de7" containerName="maas-api" Oct 29 23:08:00.738983 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.738957 4821 cpu_manager.go:414] "containerMap: removing container" podUID="d648624c-86e0-4640-8e90-048cd2c02d19" containerName="maas-api" Oct 29 23:08:00.738983 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.738961 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d648624c-86e0-4640-8e90-048cd2c02d19" containerName="maas-api" Oct 29 23:08:00.739082 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.738987 4821 cpu_manager.go:414] "containerMap: removing container" podUID="7c03888c-f9e6-4f33-95bb-e17e22d4ff54" containerName="payload-pre-processing" Oct 29 23:08:00.739082 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.738993 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c03888c-f9e6-4f33-95bb-e17e22d4ff54" containerName="payload-pre-processing" Oct 29 23:08:00.739144 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.739098 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b0089e6-032c-4d01-8557-987929bb2de7" containerName="maas-api" Oct 29 23:08:00.739144 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.739108 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="d648624c-86e0-4640-8e90-048cd2c02d19" containerName="maas-api" Oct 29 23:08:00.739144 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.739119 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c03888c-f9e6-4f33-95bb-e17e22d4ff54" containerName="payload-pre-processing" Oct 29 23:08:00.744294 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.744274 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq" Oct 29 23:08:00.765617 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.765590 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq"] Oct 29 23:08:00.773323 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.773295 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:08:00.773482 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.773359 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:08:00.847453 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.847419 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt4t5\" (UniqueName: \"kubernetes.io/projected/07614620-ede6-4242-94e2-57a89c2909f5-kube-api-access-zt4t5\") pod \"maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq\" (UID: \"07614620-ede6-4242-94e2-57a89c2909f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq" Oct 29 23:08:00.847648 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.847487 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07614620-ede6-4242-94e2-57a89c2909f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq\" (UID: \"07614620-ede6-4242-94e2-57a89c2909f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq" Oct 29 23:08:00.847763 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.847739 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07614620-ede6-4242-94e2-57a89c2909f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq\" (UID: \"07614620-ede6-4242-94e2-57a89c2909f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq" Oct 29 23:08:00.949031 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.948987 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07614620-ede6-4242-94e2-57a89c2909f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq\" (UID: \"07614620-ede6-4242-94e2-57a89c2909f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq" Oct 29 23:08:00.949031 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.949045 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt4t5\" (UniqueName: \"kubernetes.io/projected/07614620-ede6-4242-94e2-57a89c2909f5-kube-api-access-zt4t5\") pod \"maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq\" (UID: \"07614620-ede6-4242-94e2-57a89c2909f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq" Oct 29 23:08:00.949299 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.949091 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07614620-ede6-4242-94e2-57a89c2909f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq\" (UID: \"07614620-ede6-4242-94e2-57a89c2909f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq" Oct 29 23:08:00.949759 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.949726 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07614620-ede6-4242-94e2-57a89c2909f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq\" (UID: \"07614620-ede6-4242-94e2-57a89c2909f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq" Oct 29 23:08:00.951651 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.951616 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07614620-ede6-4242-94e2-57a89c2909f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq\" (UID: \"07614620-ede6-4242-94e2-57a89c2909f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq" Oct 29 23:08:00.973894 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:00.973862 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt4t5\" (UniqueName: \"kubernetes.io/projected/07614620-ede6-4242-94e2-57a89c2909f5-kube-api-access-zt4t5\") pod \"maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq\" (UID: \"07614620-ede6-4242-94e2-57a89c2909f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq" Oct 29 23:08:01.058699 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:01.058606 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq" Oct 29 23:08:01.371061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:01.371033 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq"] Oct 29 23:08:01.371472 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:08:01.371036 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07614620_ede6_4242_94e2_57a89c2909f5.slice/crio-26d303d93e299217ee31949d9bacbb175e53fc0efba893551badc8ea74a088fa WatchSource:0}: Error finding container 26d303d93e299217ee31949d9bacbb175e53fc0efba893551badc8ea74a088fa: Status 404 returned error can't find the container with id 26d303d93e299217ee31949d9bacbb175e53fc0efba893551badc8ea74a088fa Oct 29 23:08:01.947943 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:01.947832 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq" event={"ID":"07614620-ede6-4242-94e2-57a89c2909f5","Type":"ContainerStarted","Data":"30ffe8d1edb3718b7d6c7e81177a71c47663218703e7b6ff39c3d410a6852e10"} Oct 29 23:08:01.947943 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:01.947904 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq" event={"ID":"07614620-ede6-4242-94e2-57a89c2909f5","Type":"ContainerStarted","Data":"26d303d93e299217ee31949d9bacbb175e53fc0efba893551badc8ea74a088fa"} Oct 29 23:08:01.992331 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:01.992280 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq" podStartSLOduration=1.992264483 podStartE2EDuration="1.992264483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:01.990445468 +0000 UTC m=+1535.930993375" watchObservedRunningTime="2026-10-29 23:08:01.992264483 +0000 UTC m=+1535.932812397" Oct 29 23:08:02.957800 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:02.957756 4821 generic.go:356] "Generic (PLEG): container finished" podID="07614620-ede6-4242-94e2-57a89c2909f5" containerID="30ffe8d1edb3718b7d6c7e81177a71c47663218703e7b6ff39c3d410a6852e10" exitCode=0 Oct 29 23:08:02.958348 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:02.957832 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq" event={"ID":"07614620-ede6-4242-94e2-57a89c2909f5","Type":"ContainerDied","Data":"30ffe8d1edb3718b7d6c7e81177a71c47663218703e7b6ff39c3d410a6852e10"} Oct 29 23:08:04.180833 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:04.180808 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq" Oct 29 23:08:04.206050 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:04.206024 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07614620-ede6-4242-94e2-57a89c2909f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07614620-ede6-4242-94e2-57a89c2909f5-maas-api-xks-ca\") pod \"07614620-ede6-4242-94e2-57a89c2909f5\" (UID: \"07614620-ede6-4242-94e2-57a89c2909f5\") " Oct 29 23:08:04.206216 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:04.206121 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07614620-ede6-4242-94e2-57a89c2909f5-kube-api-access-zt4t5\" (UniqueName: \"kubernetes.io/projected/07614620-ede6-4242-94e2-57a89c2909f5-kube-api-access-zt4t5\") pod \"07614620-ede6-4242-94e2-57a89c2909f5\" (UID: \"07614620-ede6-4242-94e2-57a89c2909f5\") " Oct 29 23:08:04.206216 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:04.206149 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07614620-ede6-4242-94e2-57a89c2909f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07614620-ede6-4242-94e2-57a89c2909f5-maas-api-openshift-ca\") pod \"07614620-ede6-4242-94e2-57a89c2909f5\" (UID: \"07614620-ede6-4242-94e2-57a89c2909f5\") " Oct 29 23:08:04.206528 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:04.206505 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07614620-ede6-4242-94e2-57a89c2909f5-maas-api-openshift-ca" pod "07614620-ede6-4242-94e2-57a89c2909f5" (UID: "07614620-ede6-4242-94e2-57a89c2909f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:04.207235 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:04.207178 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07614620-ede6-4242-94e2-57a89c2909f5-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:04.208649 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:04.208626 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07614620-ede6-4242-94e2-57a89c2909f5-maas-api-xks-ca" pod "07614620-ede6-4242-94e2-57a89c2909f5" (UID: "07614620-ede6-4242-94e2-57a89c2909f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:08:04.208772 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:04.208729 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07614620-ede6-4242-94e2-57a89c2909f5-kube-api-access-zt4t5" pod "07614620-ede6-4242-94e2-57a89c2909f5" (UID: "07614620-ede6-4242-94e2-57a89c2909f5"). InnerVolumeSpecName "kube-api-access-zt4t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:04.308474 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:04.308444 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07614620-ede6-4242-94e2-57a89c2909f5-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:04.308474 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:04.308472 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt4t5\" (UniqueName: \"kubernetes.io/projected/07614620-ede6-4242-94e2-57a89c2909f5-kube-api-access-zt4t5\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:04.976136 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:04.976099 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq" Oct 29 23:08:04.976356 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:04.976132 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq" event={"ID":"07614620-ede6-4242-94e2-57a89c2909f5","Type":"ContainerDied","Data":"26d303d93e299217ee31949d9bacbb175e53fc0efba893551badc8ea74a088fa"} Oct 29 23:08:04.976356 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:04.976227 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26d303d93e299217ee31949d9bacbb175e53fc0efba893551badc8ea74a088fa" Oct 29 23:08:05.474787 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.474763 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:08:05.519545 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.519520 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:08:05.525795 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.525766 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-credential-socket\") pod \"914c88e0-ed28-42c6-b01d-1be633f49e4b\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " Oct 29 23:08:05.525927 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.525834 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-podinfo\") pod \"914c88e0-ed28-42c6-b01d-1be633f49e4b\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " Oct 29 23:08:05.525927 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.525883 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-data\") pod \"914c88e0-ed28-42c6-b01d-1be633f49e4b\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " Oct 29 23:08:05.526020 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.525925 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-envoy\") pod \"914c88e0-ed28-42c6-b01d-1be633f49e4b\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " Oct 29 23:08:05.526020 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.525990 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/914c88e0-ed28-42c6-b01d-1be633f49e4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914c88e0-ed28-42c6-b01d-1be633f49e4b-istiod-ca-cert\") pod \"914c88e0-ed28-42c6-b01d-1be633f49e4b\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " Oct 29 23:08:05.526111 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.526030 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-credential-socket" pod "914c88e0-ed28-42c6-b01d-1be633f49e4b" (UID: "914c88e0-ed28-42c6-b01d-1be633f49e4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:05.526111 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.526023 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-workload-socket\") pod \"914c88e0-ed28-42c6-b01d-1be633f49e4b\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " Oct 29 23:08:05.526111 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.526107 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914c88e0-ed28-42c6-b01d-1be633f49e4b-kube-api-access-xqdkg\" (UniqueName: \"kubernetes.io/projected/914c88e0-ed28-42c6-b01d-1be633f49e4b-kube-api-access-xqdkg\") pod \"914c88e0-ed28-42c6-b01d-1be633f49e4b\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " Oct 29 23:08:05.526336 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.526265 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-token\" (UniqueName: \"kubernetes.io/projected/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-token\") pod \"914c88e0-ed28-42c6-b01d-1be633f49e4b\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " Oct 29 23:08:05.526400 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.526374 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/914c88e0-ed28-42c6-b01d-1be633f49e4b-istiod-ca-cert" pod "914c88e0-ed28-42c6-b01d-1be633f49e4b" (UID: "914c88e0-ed28-42c6-b01d-1be633f49e4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:05.526641 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.526617 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-data" pod "914c88e0-ed28-42c6-b01d-1be633f49e4b" (UID: "914c88e0-ed28-42c6-b01d-1be633f49e4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:05.526803 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.526715 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-workload-socket" pod "914c88e0-ed28-42c6-b01d-1be633f49e4b" (UID: "914c88e0-ed28-42c6-b01d-1be633f49e4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:05.526901 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.526827 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-workload-certs" pod "914c88e0-ed28-42c6-b01d-1be633f49e4b" (UID: "914c88e0-ed28-42c6-b01d-1be633f49e4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:05.526986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.526651 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-workload-certs\") pod \"914c88e0-ed28-42c6-b01d-1be633f49e4b\" (UID: \"914c88e0-ed28-42c6-b01d-1be633f49e4b\") " Oct 29 23:08:05.527919 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.527897 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.528051 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.528037 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.528164 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.528142 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.528164 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.528165 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914c88e0-ed28-42c6-b01d-1be633f49e4b-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.528339 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.528177 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.528853 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.528828 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-podinfo" pod "914c88e0-ed28-42c6-b01d-1be633f49e4b" (UID: "914c88e0-ed28-42c6-b01d-1be633f49e4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:08:05.529010 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.528987 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-token" pod "914c88e0-ed28-42c6-b01d-1be633f49e4b" (UID: "914c88e0-ed28-42c6-b01d-1be633f49e4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:05.529745 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.529723 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914c88e0-ed28-42c6-b01d-1be633f49e4b-kube-api-access-xqdkg" pod "914c88e0-ed28-42c6-b01d-1be633f49e4b" (UID: "914c88e0-ed28-42c6-b01d-1be633f49e4b"). InnerVolumeSpecName "kube-api-access-xqdkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:05.530033 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.530006 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-envoy" pod "914c88e0-ed28-42c6-b01d-1be633f49e4b" (UID: "914c88e0-ed28-42c6-b01d-1be633f49e4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:05.629082 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.629043 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-workload-socket\") pod \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " Oct 29 23:08:05.629303 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.629113 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-workload-certs\") pod \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " Oct 29 23:08:05.629303 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.629243 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-token\" (UniqueName: \"kubernetes.io/projected/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-token\") pod \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " Oct 29 23:08:05.629456 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.629330 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istiod-ca-cert\") pod \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " Oct 29 23:08:05.629456 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.629351 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-workload-socket" pod "f1840a05-a8fe-40e2-b7f9-39a91f6f355f" (UID: "f1840a05-a8fe-40e2-b7f9-39a91f6f355f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:05.629456 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.629382 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-podinfo\") pod \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " Oct 29 23:08:05.629620 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.629487 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-data\") pod \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " Oct 29 23:08:05.629620 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.629486 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-workload-certs" pod "f1840a05-a8fe-40e2-b7f9-39a91f6f355f" (UID: "f1840a05-a8fe-40e2-b7f9-39a91f6f355f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:05.629736 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.629695 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istiod-ca-cert" pod "f1840a05-a8fe-40e2-b7f9-39a91f6f355f" (UID: "f1840a05-a8fe-40e2-b7f9-39a91f6f355f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:05.629843 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.629816 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-credential-socket" pod "f1840a05-a8fe-40e2-b7f9-39a91f6f355f" (UID: "f1840a05-a8fe-40e2-b7f9-39a91f6f355f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:05.629843 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.629832 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-data" pod "f1840a05-a8fe-40e2-b7f9-39a91f6f355f" (UID: "f1840a05-a8fe-40e2-b7f9-39a91f6f355f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:05.629961 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.629864 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-credential-socket\") pod \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " Oct 29 23:08:05.630021 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.629959 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-kube-api-access-zb5cg\" (UniqueName: \"kubernetes.io/projected/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-kube-api-access-zb5cg\") pod \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " Oct 29 23:08:05.630021 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.629995 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-envoy\") pod \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\" (UID: \"f1840a05-a8fe-40e2-b7f9-39a91f6f355f\") " Oct 29 23:08:05.630839 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.630790 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.630989 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.630919 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.631068 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.630993 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.631068 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.631008 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.631068 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.631021 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.631068 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.631035 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqdkg\" (UniqueName: \"kubernetes.io/projected/914c88e0-ed28-42c6-b01d-1be633f49e4b-kube-api-access-xqdkg\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.631068 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.631047 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.631068 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.631060 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914c88e0-ed28-42c6-b01d-1be633f49e4b-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.631421 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.631072 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.632445 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.632419 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-podinfo" pod "f1840a05-a8fe-40e2-b7f9-39a91f6f355f" (UID: "f1840a05-a8fe-40e2-b7f9-39a91f6f355f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:08:05.632737 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.632712 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-kube-api-access-zb5cg" pod "f1840a05-a8fe-40e2-b7f9-39a91f6f355f" (UID: "f1840a05-a8fe-40e2-b7f9-39a91f6f355f"). InnerVolumeSpecName "kube-api-access-zb5cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:05.632998 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.632974 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-envoy" pod "f1840a05-a8fe-40e2-b7f9-39a91f6f355f" (UID: "f1840a05-a8fe-40e2-b7f9-39a91f6f355f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:05.633177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.633161 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-token" pod "f1840a05-a8fe-40e2-b7f9-39a91f6f355f" (UID: "f1840a05-a8fe-40e2-b7f9-39a91f6f355f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:05.732855 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.732816 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.732855 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.732853 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb5cg\" (UniqueName: \"kubernetes.io/projected/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-kube-api-access-zb5cg\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.732855 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.732864 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.733113 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.732894 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1840a05-a8fe-40e2-b7f9-39a91f6f355f-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:05.986278 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.986245 4821 generic.go:356] "Generic (PLEG): container finished" podID="914c88e0-ed28-42c6-b01d-1be633f49e4b" containerID="c7790336fb3c251d1bc383f9d9109d8673895974bceb5234bc408391ce74a67c" exitCode=0 Oct 29 23:08:05.986477 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.986341 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" Oct 29 23:08:05.986477 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.986365 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" event={"ID":"914c88e0-ed28-42c6-b01d-1be633f49e4b","Type":"ContainerDied","Data":"c7790336fb3c251d1bc383f9d9109d8673895974bceb5234bc408391ce74a67c"} Oct 29 23:08:05.986477 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.986422 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl" event={"ID":"914c88e0-ed28-42c6-b01d-1be633f49e4b","Type":"ContainerDied","Data":"7e9df837bb74aa6af0a8a99cc1a1acb32d753db02f870e946f78af4fc9887c73"} Oct 29 23:08:05.986477 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.986453 4821 scope.go:122] "RemoveContainer" containerID="c7790336fb3c251d1bc383f9d9109d8673895974bceb5234bc408391ce74a67c" Oct 29 23:08:05.994087 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.994062 4821 generic.go:356] "Generic (PLEG): container finished" podID="f1840a05-a8fe-40e2-b7f9-39a91f6f355f" containerID="7e9cb7a0b247a2a1ce0018539a2a1ef4ba002451743af20747de0a6fa7c5c5d6" exitCode=0 Oct 29 23:08:05.994242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.994115 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" Oct 29 23:08:05.994242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.994150 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" event={"ID":"f1840a05-a8fe-40e2-b7f9-39a91f6f355f","Type":"ContainerDied","Data":"7e9cb7a0b247a2a1ce0018539a2a1ef4ba002451743af20747de0a6fa7c5c5d6"} Oct 29 23:08:05.994360 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.994243 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd" event={"ID":"f1840a05-a8fe-40e2-b7f9-39a91f6f355f","Type":"ContainerDied","Data":"4a422e96f5832d6e516a5611088b7bb6880495807d5fb408e166bc23488f6b94"} Oct 29 23:08:05.999410 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.999351 4821 scope.go:122] "RemoveContainer" containerID="c7790336fb3c251d1bc383f9d9109d8673895974bceb5234bc408391ce74a67c" Oct 29 23:08:05.999711 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:08:05.999686 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7790336fb3c251d1bc383f9d9109d8673895974bceb5234bc408391ce74a67c\": container with ID starting with c7790336fb3c251d1bc383f9d9109d8673895974bceb5234bc408391ce74a67c not found: ID does not exist" containerID="c7790336fb3c251d1bc383f9d9109d8673895974bceb5234bc408391ce74a67c" Oct 29 23:08:05.999802 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.999720 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7790336fb3c251d1bc383f9d9109d8673895974bceb5234bc408391ce74a67c"} err="failed to get container status \"c7790336fb3c251d1bc383f9d9109d8673895974bceb5234bc408391ce74a67c\": rpc error: code = NotFound desc = could not find container \"c7790336fb3c251d1bc383f9d9109d8673895974bceb5234bc408391ce74a67c\": container with ID starting with c7790336fb3c251d1bc383f9d9109d8673895974bceb5234bc408391ce74a67c not found: ID does not exist" Oct 29 23:08:05.999802 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:05.999744 4821 scope.go:122] "RemoveContainer" containerID="7e9cb7a0b247a2a1ce0018539a2a1ef4ba002451743af20747de0a6fa7c5c5d6" Oct 29 23:08:06.010701 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:06.010686 4821 scope.go:122] "RemoveContainer" containerID="dc0623074d766ff1f978853f514e08c7f9386d8f86c3c52ce1f735aba831b7be" Oct 29 23:08:06.021316 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:06.021296 4821 scope.go:122] "RemoveContainer" containerID="7e9cb7a0b247a2a1ce0018539a2a1ef4ba002451743af20747de0a6fa7c5c5d6" Oct 29 23:08:06.021597 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:08:06.021579 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e9cb7a0b247a2a1ce0018539a2a1ef4ba002451743af20747de0a6fa7c5c5d6\": container with ID starting with 7e9cb7a0b247a2a1ce0018539a2a1ef4ba002451743af20747de0a6fa7c5c5d6 not found: ID does not exist" containerID="7e9cb7a0b247a2a1ce0018539a2a1ef4ba002451743af20747de0a6fa7c5c5d6" Oct 29 23:08:06.021661 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:06.021605 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e9cb7a0b247a2a1ce0018539a2a1ef4ba002451743af20747de0a6fa7c5c5d6"} err="failed to get container status \"7e9cb7a0b247a2a1ce0018539a2a1ef4ba002451743af20747de0a6fa7c5c5d6\": rpc error: code = NotFound desc = could not find container \"7e9cb7a0b247a2a1ce0018539a2a1ef4ba002451743af20747de0a6fa7c5c5d6\": container with ID starting with 7e9cb7a0b247a2a1ce0018539a2a1ef4ba002451743af20747de0a6fa7c5c5d6 not found: ID does not exist" Oct 29 23:08:06.021661 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:06.021623 4821 scope.go:122] "RemoveContainer" containerID="dc0623074d766ff1f978853f514e08c7f9386d8f86c3c52ce1f735aba831b7be" Oct 29 23:08:06.021862 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:08:06.021845 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc0623074d766ff1f978853f514e08c7f9386d8f86c3c52ce1f735aba831b7be\": container with ID starting with dc0623074d766ff1f978853f514e08c7f9386d8f86c3c52ce1f735aba831b7be not found: ID does not exist" containerID="dc0623074d766ff1f978853f514e08c7f9386d8f86c3c52ce1f735aba831b7be" Oct 29 23:08:06.021908 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:06.021867 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc0623074d766ff1f978853f514e08c7f9386d8f86c3c52ce1f735aba831b7be"} err="failed to get container status \"dc0623074d766ff1f978853f514e08c7f9386d8f86c3c52ce1f735aba831b7be\": rpc error: code = NotFound desc = could not find container \"dc0623074d766ff1f978853f514e08c7f9386d8f86c3c52ce1f735aba831b7be\": container with ID starting with dc0623074d766ff1f978853f514e08c7f9386d8f86c3c52ce1f735aba831b7be not found: ID does not exist" Oct 29 23:08:06.062006 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:06.061979 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl"] Oct 29 23:08:06.072524 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:06.072501 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-rlqbl"] Oct 29 23:08:06.099617 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:06.099573 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd"] Oct 29 23:08:06.103594 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:06.103569 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7950-openshift-default-6cbd77447f-8wrsd"] Oct 29 23:08:06.170430 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:06.170386 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="914c88e0-ed28-42c6-b01d-1be633f49e4b" path="/var/lib/kubelet/pods/914c88e0-ed28-42c6-b01d-1be633f49e4b/volumes" Oct 29 23:08:06.170940 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:06.170925 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1840a05-a8fe-40e2-b7f9-39a91f6f355f" path="/var/lib/kubelet/pods/f1840a05-a8fe-40e2-b7f9-39a91f6f355f/volumes" Oct 29 23:08:06.850370 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:06.850337 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp"] Oct 29 23:08:06.850828 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:06.850542 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" podUID="7874a224-3fab-4fee-90db-89b753ed938f" containerName="istio-proxy" containerID="cri-o://af11dc8eb42c2bc59c9001037e0df69864aaccb9b5ca5eb4b39eb4b63f125893" gracePeriod=30 Oct 29 23:08:07.894663 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:07.894623 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv"] Oct 29 23:08:07.895445 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:07.895420 4821 cpu_manager.go:414] "containerMap: removing container" podUID="f1840a05-a8fe-40e2-b7f9-39a91f6f355f" containerName="istio-proxy" Oct 29 23:08:07.895445 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:07.895444 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1840a05-a8fe-40e2-b7f9-39a91f6f355f" containerName="istio-proxy" Oct 29 23:08:07.895633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:07.895505 4821 cpu_manager.go:414] "containerMap: removing container" podUID="07614620-ede6-4242-94e2-57a89c2909f5" containerName="revoke-keys" Oct 29 23:08:07.895633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:07.895513 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07614620-ede6-4242-94e2-57a89c2909f5" containerName="revoke-keys" Oct 29 23:08:07.895633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:07.895550 4821 cpu_manager.go:414] "containerMap: removing container" podUID="914c88e0-ed28-42c6-b01d-1be633f49e4b" containerName="istio-proxy" Oct 29 23:08:07.895633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:07.895559 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="914c88e0-ed28-42c6-b01d-1be633f49e4b" containerName="istio-proxy" Oct 29 23:08:07.895633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:07.895568 4821 cpu_manager.go:414] "containerMap: removing container" podUID="f1840a05-a8fe-40e2-b7f9-39a91f6f355f" containerName="istio-proxy" Oct 29 23:08:07.895633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:07.895574 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1840a05-a8fe-40e2-b7f9-39a91f6f355f" containerName="istio-proxy" Oct 29 23:08:07.895929 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:07.895724 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="914c88e0-ed28-42c6-b01d-1be633f49e4b" containerName="istio-proxy" Oct 29 23:08:07.895929 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:07.895738 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1840a05-a8fe-40e2-b7f9-39a91f6f355f" containerName="istio-proxy" Oct 29 23:08:07.895929 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:07.895749 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1840a05-a8fe-40e2-b7f9-39a91f6f355f" containerName="istio-proxy" Oct 29 23:08:07.895929 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:07.895765 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="07614620-ede6-4242-94e2-57a89c2909f5" containerName="revoke-keys" Oct 29 23:08:07.900838 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:07.900818 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv" Oct 29 23:08:07.924430 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:07.924393 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv"] Oct 29 23:08:07.929074 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:07.929049 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:08:07.929338 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:07.929316 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:08:07.965783 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:07.965745 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8000f02e-22df-47a9-ac72-5d245e52218c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv\" (UID: \"8000f02e-22df-47a9-ac72-5d245e52218c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv" Oct 29 23:08:07.966000 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:07.965981 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29fbl\" (UniqueName: \"kubernetes.io/projected/8000f02e-22df-47a9-ac72-5d245e52218c-kube-api-access-29fbl\") pod \"maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv\" (UID: \"8000f02e-22df-47a9-ac72-5d245e52218c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv" Oct 29 23:08:07.966151 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:07.966133 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8000f02e-22df-47a9-ac72-5d245e52218c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv\" (UID: \"8000f02e-22df-47a9-ac72-5d245e52218c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv" Oct 29 23:08:08.067928 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:08.067891 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8000f02e-22df-47a9-ac72-5d245e52218c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv\" (UID: \"8000f02e-22df-47a9-ac72-5d245e52218c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv" Oct 29 23:08:08.068098 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:08.068043 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29fbl\" (UniqueName: \"kubernetes.io/projected/8000f02e-22df-47a9-ac72-5d245e52218c-kube-api-access-29fbl\") pod \"maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv\" (UID: \"8000f02e-22df-47a9-ac72-5d245e52218c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv" Oct 29 23:08:08.068190 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:08.068158 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8000f02e-22df-47a9-ac72-5d245e52218c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv\" (UID: \"8000f02e-22df-47a9-ac72-5d245e52218c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv" Oct 29 23:08:08.068609 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:08.068589 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8000f02e-22df-47a9-ac72-5d245e52218c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv\" (UID: \"8000f02e-22df-47a9-ac72-5d245e52218c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv" Oct 29 23:08:08.070637 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:08.070613 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8000f02e-22df-47a9-ac72-5d245e52218c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv\" (UID: \"8000f02e-22df-47a9-ac72-5d245e52218c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv" Oct 29 23:08:08.092787 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:08.092759 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29fbl\" (UniqueName: \"kubernetes.io/projected/8000f02e-22df-47a9-ac72-5d245e52218c-kube-api-access-29fbl\") pod \"maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv\" (UID: \"8000f02e-22df-47a9-ac72-5d245e52218c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv" Oct 29 23:08:08.227409 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:08.227307 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv" Oct 29 23:08:08.536541 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:08:08.536500 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8000f02e_22df_47a9_ac72_5d245e52218c.slice/crio-6db324ea62a927239326a26a32164161f341f7aa57a948c62f2cee0e2818eb52 WatchSource:0}: Error finding container 6db324ea62a927239326a26a32164161f341f7aa57a948c62f2cee0e2818eb52: Status 404 returned error can't find the container with id 6db324ea62a927239326a26a32164161f341f7aa57a948c62f2cee0e2818eb52 Oct 29 23:08:08.536683 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:08.536541 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv"] Oct 29 23:08:09.023061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:09.023022 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv" event={"ID":"8000f02e-22df-47a9-ac72-5d245e52218c","Type":"ContainerStarted","Data":"410ecacab497ead8a0fd84a38d3880fdb46ceaf99c25f80661434fa0a5ec347b"} Oct 29 23:08:09.023484 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:09.023082 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv" event={"ID":"8000f02e-22df-47a9-ac72-5d245e52218c","Type":"ContainerStarted","Data":"6db324ea62a927239326a26a32164161f341f7aa57a948c62f2cee0e2818eb52"} Oct 29 23:08:09.067944 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:09.067890 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv" podStartSLOduration=2.067875289 podStartE2EDuration="2.067875289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:09.064247866 +0000 UTC m=+1543.004795779" watchObservedRunningTime="2026-10-29 23:08:09.067875289 +0000 UTC m=+1543.008423200" Oct 29 23:08:09.770291 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:09.770093 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" podUID="0a560010-b749-4e0d-875c-4b5d880410ef" containerName="main" containerID="cri-o://88e5d6e180ce3947b2d2a842a86e51aa2def76b0962b9e1acfd78620b9270b15" gracePeriod=45 Oct 29 23:08:10.034773 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:10.034691 4821 generic.go:356] "Generic (PLEG): container finished" podID="0a560010-b749-4e0d-875c-4b5d880410ef" containerID="88e5d6e180ce3947b2d2a842a86e51aa2def76b0962b9e1acfd78620b9270b15" exitCode=0 Oct 29 23:08:10.034773 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:10.034761 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" event={"ID":"0a560010-b749-4e0d-875c-4b5d880410ef","Type":"ContainerDied","Data":"88e5d6e180ce3947b2d2a842a86e51aa2def76b0962b9e1acfd78620b9270b15"} Oct 29 23:08:10.036739 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:10.036710 4821 generic.go:356] "Generic (PLEG): container finished" podID="8000f02e-22df-47a9-ac72-5d245e52218c" containerID="410ecacab497ead8a0fd84a38d3880fdb46ceaf99c25f80661434fa0a5ec347b" exitCode=0 Oct 29 23:08:10.036873 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:10.036751 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv" event={"ID":"8000f02e-22df-47a9-ac72-5d245e52218c","Type":"ContainerDied","Data":"410ecacab497ead8a0fd84a38d3880fdb46ceaf99c25f80661434fa0a5ec347b"} Oct 29 23:08:10.068873 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:10.068848 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:08:10.200661 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:10.200620 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a560010-b749-4e0d-875c-4b5d880410ef-kube-api-access-hns86\" (UniqueName: \"kubernetes.io/projected/0a560010-b749-4e0d-875c-4b5d880410ef-kube-api-access-hns86\") pod \"0a560010-b749-4e0d-875c-4b5d880410ef\" (UID: \"0a560010-b749-4e0d-875c-4b5d880410ef\") " Oct 29 23:08:10.200866 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:10.200710 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-dshm\") pod \"0a560010-b749-4e0d-875c-4b5d880410ef\" (UID: \"0a560010-b749-4e0d-875c-4b5d880410ef\") " Oct 29 23:08:10.200866 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:10.200738 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-model-cache\") pod \"0a560010-b749-4e0d-875c-4b5d880410ef\" (UID: \"0a560010-b749-4e0d-875c-4b5d880410ef\") " Oct 29 23:08:10.200866 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:10.200774 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-tmp-dir\") pod \"0a560010-b749-4e0d-875c-4b5d880410ef\" (UID: \"0a560010-b749-4e0d-875c-4b5d880410ef\") " Oct 29 23:08:10.200866 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:10.200792 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a560010-b749-4e0d-875c-4b5d880410ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a560010-b749-4e0d-875c-4b5d880410ef-tls-certs\") pod \"0a560010-b749-4e0d-875c-4b5d880410ef\" (UID: \"0a560010-b749-4e0d-875c-4b5d880410ef\") " Oct 29 23:08:10.201090 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:10.200880 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-home\" (UniqueName: \"kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-home\") pod \"0a560010-b749-4e0d-875c-4b5d880410ef\" (UID: \"0a560010-b749-4e0d-875c-4b5d880410ef\") " Oct 29 23:08:10.201090 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:10.201037 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-model-cache" pod "0a560010-b749-4e0d-875c-4b5d880410ef" (UID: "0a560010-b749-4e0d-875c-4b5d880410ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:10.201090 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:10.201057 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-tmp-dir" pod "0a560010-b749-4e0d-875c-4b5d880410ef" (UID: "0a560010-b749-4e0d-875c-4b5d880410ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:10.201290 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:10.201179 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-home" pod "0a560010-b749-4e0d-875c-4b5d880410ef" (UID: "0a560010-b749-4e0d-875c-4b5d880410ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:10.201570 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:10.201549 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:10.201570 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:10.201571 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:10.201723 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:10.201583 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:10.203080 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:10.203061 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a560010-b749-4e0d-875c-4b5d880410ef-tls-certs" pod "0a560010-b749-4e0d-875c-4b5d880410ef" (UID: "0a560010-b749-4e0d-875c-4b5d880410ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:08:10.203143 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:10.203082 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-dshm" pod "0a560010-b749-4e0d-875c-4b5d880410ef" (UID: "0a560010-b749-4e0d-875c-4b5d880410ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:10.203232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:10.203212 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a560010-b749-4e0d-875c-4b5d880410ef-kube-api-access-hns86" pod "0a560010-b749-4e0d-875c-4b5d880410ef" (UID: "0a560010-b749-4e0d-875c-4b5d880410ef"). InnerVolumeSpecName "kube-api-access-hns86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:10.303558 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:10.303447 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hns86\" (UniqueName: \"kubernetes.io/projected/0a560010-b749-4e0d-875c-4b5d880410ef-kube-api-access-hns86\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:10.303558 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:10.303491 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a560010-b749-4e0d-875c-4b5d880410ef-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:10.303558 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:10.303510 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a560010-b749-4e0d-875c-4b5d880410ef-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:10.980883 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:10.980844 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" podUID="0a560010-b749-4e0d-875c-4b5d880410ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.155:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:11.044625 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:11.044587 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:08:11.045035 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:11.044657 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" podUID="7874a224-3fab-4fee-90db-89b753ed938f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:08:11.046372 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:11.046343 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" Oct 29 23:08:11.046372 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:11.046357 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4" event={"ID":"0a560010-b749-4e0d-875c-4b5d880410ef","Type":"ContainerDied","Data":"428822774d3a01f5bcf2830f005574930854e6df14115034276c8a9d020bcfe4"} Oct 29 23:08:11.046583 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:11.046413 4821 scope.go:122] "RemoveContainer" containerID="88e5d6e180ce3947b2d2a842a86e51aa2def76b0962b9e1acfd78620b9270b15" Oct 29 23:08:11.104357 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:11.104325 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4"] Oct 29 23:08:11.109058 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:11.109030 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-d67deb/praxis-route-d67deb-kserve-5984778997-dx2v4"] Oct 29 23:08:11.235294 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:11.235233 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv" Oct 29 23:08:11.319472 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:11.319435 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8000f02e-22df-47a9-ac72-5d245e52218c-kube-api-access-29fbl\" (UniqueName: \"kubernetes.io/projected/8000f02e-22df-47a9-ac72-5d245e52218c-kube-api-access-29fbl\") pod \"8000f02e-22df-47a9-ac72-5d245e52218c\" (UID: \"8000f02e-22df-47a9-ac72-5d245e52218c\") " Oct 29 23:08:11.319678 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:11.319501 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8000f02e-22df-47a9-ac72-5d245e52218c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8000f02e-22df-47a9-ac72-5d245e52218c-maas-api-xks-ca\") pod \"8000f02e-22df-47a9-ac72-5d245e52218c\" (UID: \"8000f02e-22df-47a9-ac72-5d245e52218c\") " Oct 29 23:08:11.319756 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:11.319720 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8000f02e-22df-47a9-ac72-5d245e52218c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8000f02e-22df-47a9-ac72-5d245e52218c-maas-api-openshift-ca\") pod \"8000f02e-22df-47a9-ac72-5d245e52218c\" (UID: \"8000f02e-22df-47a9-ac72-5d245e52218c\") " Oct 29 23:08:11.320229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:11.320134 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8000f02e-22df-47a9-ac72-5d245e52218c-maas-api-openshift-ca" pod "8000f02e-22df-47a9-ac72-5d245e52218c" (UID: "8000f02e-22df-47a9-ac72-5d245e52218c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:11.321037 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:11.320773 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8000f02e-22df-47a9-ac72-5d245e52218c-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:11.322019 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:11.321998 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8000f02e-22df-47a9-ac72-5d245e52218c-maas-api-xks-ca" pod "8000f02e-22df-47a9-ac72-5d245e52218c" (UID: "8000f02e-22df-47a9-ac72-5d245e52218c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:08:11.322374 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:11.322359 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8000f02e-22df-47a9-ac72-5d245e52218c-kube-api-access-29fbl" pod "8000f02e-22df-47a9-ac72-5d245e52218c" (UID: "8000f02e-22df-47a9-ac72-5d245e52218c"). InnerVolumeSpecName "kube-api-access-29fbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:11.422136 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:11.422094 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29fbl\" (UniqueName: \"kubernetes.io/projected/8000f02e-22df-47a9-ac72-5d245e52218c-kube-api-access-29fbl\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:11.422136 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:11.422134 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8000f02e-22df-47a9-ac72-5d245e52218c-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:12.059893 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.059865 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv" Oct 29 23:08:12.059893 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.059870 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv" event={"ID":"8000f02e-22df-47a9-ac72-5d245e52218c","Type":"ContainerDied","Data":"6db324ea62a927239326a26a32164161f341f7aa57a948c62f2cee0e2818eb52"} Oct 29 23:08:12.060422 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.059924 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6db324ea62a927239326a26a32164161f341f7aa57a948c62f2cee0e2818eb52" Oct 29 23:08:12.062569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.062540 4821 generic.go:356] "Generic (PLEG): container finished" podID="7874a224-3fab-4fee-90db-89b753ed938f" containerID="af11dc8eb42c2bc59c9001037e0df69864aaccb9b5ca5eb4b39eb4b63f125893" exitCode=0 Oct 29 23:08:12.062721 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.062588 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" event={"ID":"7874a224-3fab-4fee-90db-89b753ed938f","Type":"ContainerDied","Data":"af11dc8eb42c2bc59c9001037e0df69864aaccb9b5ca5eb4b39eb4b63f125893"} Oct 29 23:08:12.062721 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.062625 4821 scope.go:122] "RemoveContainer" containerID="b5fdd38111e89a63a92f4d03e43c5fd334d1d0f893220d07aa97b24ee00462bf" Oct 29 23:08:12.139329 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.139306 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 23:08:12.171149 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.171118 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a560010-b749-4e0d-875c-4b5d880410ef" path="/var/lib/kubelet/pods/0a560010-b749-4e0d-875c-4b5d880410ef/volumes" Oct 29 23:08:12.232622 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.232590 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-istio-envoy\") pod \"7874a224-3fab-4fee-90db-89b753ed938f\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " Oct 29 23:08:12.232779 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.232679 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-workload-socket\") pod \"7874a224-3fab-4fee-90db-89b753ed938f\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " Oct 29 23:08:12.232779 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.232720 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7874a224-3fab-4fee-90db-89b753ed938f-kube-api-access-p9d2n\" (UniqueName: \"kubernetes.io/projected/7874a224-3fab-4fee-90db-89b753ed938f-kube-api-access-p9d2n\") pod \"7874a224-3fab-4fee-90db-89b753ed938f\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " Oct 29 23:08:12.232779 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.232770 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-istio-data\") pod \"7874a224-3fab-4fee-90db-89b753ed938f\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " Oct 29 23:08:12.233086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.232834 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-workload-certs\") pod \"7874a224-3fab-4fee-90db-89b753ed938f\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " Oct 29 23:08:12.233086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.232872 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7874a224-3fab-4fee-90db-89b753ed938f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7874a224-3fab-4fee-90db-89b753ed938f-istio-podinfo\") pod \"7874a224-3fab-4fee-90db-89b753ed938f\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " Oct 29 23:08:12.233086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.232973 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-credential-socket\") pod \"7874a224-3fab-4fee-90db-89b753ed938f\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " Oct 29 23:08:12.233086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.233066 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7874a224-3fab-4fee-90db-89b753ed938f-istio-token\" (UniqueName: \"kubernetes.io/projected/7874a224-3fab-4fee-90db-89b753ed938f-istio-token\") pod \"7874a224-3fab-4fee-90db-89b753ed938f\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " Oct 29 23:08:12.233361 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.233138 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7874a224-3fab-4fee-90db-89b753ed938f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7874a224-3fab-4fee-90db-89b753ed938f-istiod-ca-cert\") pod \"7874a224-3fab-4fee-90db-89b753ed938f\" (UID: \"7874a224-3fab-4fee-90db-89b753ed938f\") " Oct 29 23:08:12.233361 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.233180 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-istio-data" pod "7874a224-3fab-4fee-90db-89b753ed938f" (UID: "7874a224-3fab-4fee-90db-89b753ed938f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:12.233539 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.233379 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-credential-socket" pod "7874a224-3fab-4fee-90db-89b753ed938f" (UID: "7874a224-3fab-4fee-90db-89b753ed938f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:12.233802 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.233648 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-workload-certs" pod "7874a224-3fab-4fee-90db-89b753ed938f" (UID: "7874a224-3fab-4fee-90db-89b753ed938f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:12.233802 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.233713 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-workload-socket" pod "7874a224-3fab-4fee-90db-89b753ed938f" (UID: "7874a224-3fab-4fee-90db-89b753ed938f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:12.233802 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.233756 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7874a224-3fab-4fee-90db-89b753ed938f-istiod-ca-cert" pod "7874a224-3fab-4fee-90db-89b753ed938f" (UID: "7874a224-3fab-4fee-90db-89b753ed938f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:12.234404 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.234361 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:12.234404 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.234388 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:12.234404 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.234403 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:12.234635 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.234415 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:12.234635 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.234429 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7874a224-3fab-4fee-90db-89b753ed938f-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:12.236833 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.236809 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-istio-envoy" pod "7874a224-3fab-4fee-90db-89b753ed938f" (UID: "7874a224-3fab-4fee-90db-89b753ed938f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:12.237315 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.237283 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7874a224-3fab-4fee-90db-89b753ed938f-kube-api-access-p9d2n" pod "7874a224-3fab-4fee-90db-89b753ed938f" (UID: "7874a224-3fab-4fee-90db-89b753ed938f"). InnerVolumeSpecName "kube-api-access-p9d2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:12.237315 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.237302 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7874a224-3fab-4fee-90db-89b753ed938f-istio-podinfo" pod "7874a224-3fab-4fee-90db-89b753ed938f" (UID: "7874a224-3fab-4fee-90db-89b753ed938f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:08:12.237469 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.237339 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7874a224-3fab-4fee-90db-89b753ed938f-istio-token" pod "7874a224-3fab-4fee-90db-89b753ed938f" (UID: "7874a224-3fab-4fee-90db-89b753ed938f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:12.335634 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.335601 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7874a224-3fab-4fee-90db-89b753ed938f-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:12.335634 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.335629 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7874a224-3fab-4fee-90db-89b753ed938f-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:12.335634 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.335639 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9d2n\" (UniqueName: \"kubernetes.io/projected/7874a224-3fab-4fee-90db-89b753ed938f-kube-api-access-p9d2n\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:12.335872 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:12.335649 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7874a224-3fab-4fee-90db-89b753ed938f-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:13.072962 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:13.072932 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" Oct 29 23:08:13.073522 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:13.072972 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp" event={"ID":"7874a224-3fab-4fee-90db-89b753ed938f","Type":"ContainerDied","Data":"f109d3915487834276eb9fc6b3263a1108513e51768f06b0591929279ae7620e"} Oct 29 23:08:13.073522 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:13.073031 4821 scope.go:122] "RemoveContainer" containerID="af11dc8eb42c2bc59c9001037e0df69864aaccb9b5ca5eb4b39eb4b63f125893" Oct 29 23:08:13.131827 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:13.131798 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp"] Oct 29 23:08:13.144547 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:13.144518 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-lncjp"] Oct 29 23:08:14.172626 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:14.172591 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7874a224-3fab-4fee-90db-89b753ed938f" path="/var/lib/kubelet/pods/7874a224-3fab-4fee-90db-89b753ed938f/volumes" Oct 29 23:08:16.645517 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.645479 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4cbda27-vpx9w"] Oct 29 23:08:16.646362 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.646340 4821 cpu_manager.go:414] "containerMap: removing container" podUID="0a560010-b749-4e0d-875c-4b5d880410ef" containerName="main" Oct 29 23:08:16.646362 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.646361 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a560010-b749-4e0d-875c-4b5d880410ef" containerName="main" Oct 29 23:08:16.646561 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.646377 4821 cpu_manager.go:414] "containerMap: removing container" podUID="8000f02e-22df-47a9-ac72-5d245e52218c" containerName="revoke-keys" Oct 29 23:08:16.646561 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.646384 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8000f02e-22df-47a9-ac72-5d245e52218c" containerName="revoke-keys" Oct 29 23:08:16.646561 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.646399 4821 cpu_manager.go:414] "containerMap: removing container" podUID="7874a224-3fab-4fee-90db-89b753ed938f" containerName="istio-proxy" Oct 29 23:08:16.646561 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.646406 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7874a224-3fab-4fee-90db-89b753ed938f" containerName="istio-proxy" Oct 29 23:08:16.646561 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.646445 4821 cpu_manager.go:414] "containerMap: removing container" podUID="7874a224-3fab-4fee-90db-89b753ed938f" containerName="istio-proxy" Oct 29 23:08:16.646561 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.646454 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7874a224-3fab-4fee-90db-89b753ed938f" containerName="istio-proxy" Oct 29 23:08:16.646872 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.646602 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a560010-b749-4e0d-875c-4b5d880410ef" containerName="main" Oct 29 23:08:16.646872 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.646618 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="8000f02e-22df-47a9-ac72-5d245e52218c" containerName="revoke-keys" Oct 29 23:08:16.646872 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.646628 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="7874a224-3fab-4fee-90db-89b753ed938f" containerName="istio-proxy" Oct 29 23:08:16.646872 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.646641 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="7874a224-3fab-4fee-90db-89b753ed938f" containerName="istio-proxy" Oct 29 23:08:16.650343 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.650323 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cbda27-vpx9w" Oct 29 23:08:16.680298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.680272 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:08:16.680514 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.680491 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:08:16.688247 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.687684 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4cbda27-vpx9w"] Oct 29 23:08:16.714305 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.714266 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52620065-ff2f-4acc-b33d-3e283eb5076e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4cbda27-vpx9w\" (UID: \"52620065-ff2f-4acc-b33d-3e283eb5076e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cbda27-vpx9w" Oct 29 23:08:16.714468 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.714367 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmcxh\" (UniqueName: \"kubernetes.io/projected/52620065-ff2f-4acc-b33d-3e283eb5076e-kube-api-access-lmcxh\") pod \"maas-api-revoke-sub-b4cbda27-vpx9w\" (UID: \"52620065-ff2f-4acc-b33d-3e283eb5076e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cbda27-vpx9w" Oct 29 23:08:16.714468 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.714428 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52620065-ff2f-4acc-b33d-3e283eb5076e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4cbda27-vpx9w\" (UID: \"52620065-ff2f-4acc-b33d-3e283eb5076e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cbda27-vpx9w" Oct 29 23:08:16.815584 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.815544 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmcxh\" (UniqueName: \"kubernetes.io/projected/52620065-ff2f-4acc-b33d-3e283eb5076e-kube-api-access-lmcxh\") pod \"maas-api-revoke-sub-b4cbda27-vpx9w\" (UID: \"52620065-ff2f-4acc-b33d-3e283eb5076e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cbda27-vpx9w" Oct 29 23:08:16.815760 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.815635 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52620065-ff2f-4acc-b33d-3e283eb5076e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4cbda27-vpx9w\" (UID: \"52620065-ff2f-4acc-b33d-3e283eb5076e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cbda27-vpx9w" Oct 29 23:08:16.815864 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.815851 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52620065-ff2f-4acc-b33d-3e283eb5076e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4cbda27-vpx9w\" (UID: \"52620065-ff2f-4acc-b33d-3e283eb5076e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cbda27-vpx9w" Oct 29 23:08:16.816333 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.816313 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52620065-ff2f-4acc-b33d-3e283eb5076e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4cbda27-vpx9w\" (UID: \"52620065-ff2f-4acc-b33d-3e283eb5076e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cbda27-vpx9w" Oct 29 23:08:16.818363 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.818341 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52620065-ff2f-4acc-b33d-3e283eb5076e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4cbda27-vpx9w\" (UID: \"52620065-ff2f-4acc-b33d-3e283eb5076e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cbda27-vpx9w" Oct 29 23:08:16.840071 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.840038 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmcxh\" (UniqueName: \"kubernetes.io/projected/52620065-ff2f-4acc-b33d-3e283eb5076e-kube-api-access-lmcxh\") pod \"maas-api-revoke-sub-b4cbda27-vpx9w\" (UID: \"52620065-ff2f-4acc-b33d-3e283eb5076e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cbda27-vpx9w" Oct 29 23:08:16.965620 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:16.965535 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cbda27-vpx9w" Oct 29 23:08:17.274665 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:17.274642 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4cbda27-vpx9w"] Oct 29 23:08:17.275017 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:08:17.274995 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52620065_ff2f_4acc_b33d_3e283eb5076e.slice/crio-83eb46192dc789861fa94f278907a1c00c5ddfeb2be08bc7ff88954a0c0efbbd WatchSource:0}: Error finding container 83eb46192dc789861fa94f278907a1c00c5ddfeb2be08bc7ff88954a0c0efbbd: Status 404 returned error can't find the container with id 83eb46192dc789861fa94f278907a1c00c5ddfeb2be08bc7ff88954a0c0efbbd Oct 29 23:08:18.121025 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:18.120986 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cbda27-vpx9w" event={"ID":"52620065-ff2f-4acc-b33d-3e283eb5076e","Type":"ContainerStarted","Data":"b418af0f31a65920f450e2c7d16201ea382144c97ab3b234632a12800fff8704"} Oct 29 23:08:18.121421 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:18.121072 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cbda27-vpx9w" event={"ID":"52620065-ff2f-4acc-b33d-3e283eb5076e","Type":"ContainerStarted","Data":"83eb46192dc789861fa94f278907a1c00c5ddfeb2be08bc7ff88954a0c0efbbd"} Oct 29 23:08:18.165734 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:18.165683 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cbda27-vpx9w" podStartSLOduration=2.165669869 podStartE2EDuration="2.165669869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:18.163262251 +0000 UTC m=+1552.103810162" watchObservedRunningTime="2026-10-29 23:08:18.165669869 +0000 UTC m=+1552.106217778" Oct 29 23:08:18.334004 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:18.333972 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475"] Oct 29 23:08:18.368938 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:18.368906 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8"] Oct 29 23:08:19.140891 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:19.140845 4821 generic.go:356] "Generic (PLEG): container finished" podID="52620065-ff2f-4acc-b33d-3e283eb5076e" containerID="b418af0f31a65920f450e2c7d16201ea382144c97ab3b234632a12800fff8704" exitCode=0 Oct 29 23:08:19.141328 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:19.140981 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cbda27-vpx9w" event={"ID":"52620065-ff2f-4acc-b33d-3e283eb5076e","Type":"ContainerDied","Data":"b418af0f31a65920f450e2c7d16201ea382144c97ab3b234632a12800fff8704"} Oct 29 23:08:20.320180 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:20.320153 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cbda27-vpx9w" Oct 29 23:08:20.390570 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:20.390520 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52620065-ff2f-4acc-b33d-3e283eb5076e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52620065-ff2f-4acc-b33d-3e283eb5076e-maas-api-xks-ca\") pod \"52620065-ff2f-4acc-b33d-3e283eb5076e\" (UID: \"52620065-ff2f-4acc-b33d-3e283eb5076e\") " Oct 29 23:08:20.390773 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:20.390753 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52620065-ff2f-4acc-b33d-3e283eb5076e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52620065-ff2f-4acc-b33d-3e283eb5076e-maas-api-openshift-ca\") pod \"52620065-ff2f-4acc-b33d-3e283eb5076e\" (UID: \"52620065-ff2f-4acc-b33d-3e283eb5076e\") " Oct 29 23:08:20.390837 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:20.390821 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52620065-ff2f-4acc-b33d-3e283eb5076e-kube-api-access-lmcxh\" (UniqueName: \"kubernetes.io/projected/52620065-ff2f-4acc-b33d-3e283eb5076e-kube-api-access-lmcxh\") pod \"52620065-ff2f-4acc-b33d-3e283eb5076e\" (UID: \"52620065-ff2f-4acc-b33d-3e283eb5076e\") " Oct 29 23:08:20.391750 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:20.391711 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52620065-ff2f-4acc-b33d-3e283eb5076e-maas-api-openshift-ca" pod "52620065-ff2f-4acc-b33d-3e283eb5076e" (UID: "52620065-ff2f-4acc-b33d-3e283eb5076e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:20.393033 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:20.393007 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52620065-ff2f-4acc-b33d-3e283eb5076e-maas-api-xks-ca" pod "52620065-ff2f-4acc-b33d-3e283eb5076e" (UID: "52620065-ff2f-4acc-b33d-3e283eb5076e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:08:20.393467 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:20.393447 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52620065-ff2f-4acc-b33d-3e283eb5076e-kube-api-access-lmcxh" pod "52620065-ff2f-4acc-b33d-3e283eb5076e" (UID: "52620065-ff2f-4acc-b33d-3e283eb5076e"). InnerVolumeSpecName "kube-api-access-lmcxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:20.492892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:20.492860 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52620065-ff2f-4acc-b33d-3e283eb5076e-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:20.492892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:20.492888 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmcxh\" (UniqueName: \"kubernetes.io/projected/52620065-ff2f-4acc-b33d-3e283eb5076e-kube-api-access-lmcxh\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:20.492892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:20.492899 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52620065-ff2f-4acc-b33d-3e283eb5076e-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:21.168248 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:21.167598 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cbda27-vpx9w" Oct 29 23:08:21.168248 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:21.167638 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4cbda27-vpx9w" event={"ID":"52620065-ff2f-4acc-b33d-3e283eb5076e","Type":"ContainerDied","Data":"83eb46192dc789861fa94f278907a1c00c5ddfeb2be08bc7ff88954a0c0efbbd"} Oct 29 23:08:21.168248 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:21.167681 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83eb46192dc789861fa94f278907a1c00c5ddfeb2be08bc7ff88954a0c0efbbd" Oct 29 23:08:23.334633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:23.334575 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475" podUID="ca53539d-ef76-4a59-8b01-815fe110a113" containerName="payload-processing" containerID="cri-o://fe248149a0611f21bc014106d80ecc0bb6000bc628ff0057852d079b457dc33e" gracePeriod=25 Oct 29 23:08:23.369715 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:23.369658 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8" podUID="bee57d28-4336-4792-aeeb-044ee8338a9b" containerName="payload-pre-processing" containerID="cri-o://b7a212ae818443ffc011b3bb557c5741b863678db7d8d5f3e844357d86e8779c" gracePeriod=25 Oct 29 23:08:23.649587 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:23.649554 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475" Oct 29 23:08:23.704006 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:23.703975 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8" Oct 29 23:08:23.753422 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:23.753266 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca53539d-ef76-4a59-8b01-815fe110a113-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca53539d-ef76-4a59-8b01-815fe110a113-plugins-config-volume\") pod \"ca53539d-ef76-4a59-8b01-815fe110a113\" (UID: \"ca53539d-ef76-4a59-8b01-815fe110a113\") " Oct 29 23:08:23.753612 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:23.753505 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca53539d-ef76-4a59-8b01-815fe110a113-kube-api-access-ckpq7\" (UniqueName: \"kubernetes.io/projected/ca53539d-ef76-4a59-8b01-815fe110a113-kube-api-access-ckpq7\") pod \"ca53539d-ef76-4a59-8b01-815fe110a113\" (UID: \"ca53539d-ef76-4a59-8b01-815fe110a113\") " Oct 29 23:08:23.753831 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:23.753808 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca53539d-ef76-4a59-8b01-815fe110a113-plugins-config-volume" pod "ca53539d-ef76-4a59-8b01-815fe110a113" (UID: "ca53539d-ef76-4a59-8b01-815fe110a113"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:23.754110 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:23.754088 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca53539d-ef76-4a59-8b01-815fe110a113-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:23.756192 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:23.756162 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca53539d-ef76-4a59-8b01-815fe110a113-kube-api-access-ckpq7" pod "ca53539d-ef76-4a59-8b01-815fe110a113" (UID: "ca53539d-ef76-4a59-8b01-815fe110a113"). InnerVolumeSpecName "kube-api-access-ckpq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:23.854854 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:23.854821 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bee57d28-4336-4792-aeeb-044ee8338a9b-kube-api-access-kf8ml\" (UniqueName: \"kubernetes.io/projected/bee57d28-4336-4792-aeeb-044ee8338a9b-kube-api-access-kf8ml\") pod \"bee57d28-4336-4792-aeeb-044ee8338a9b\" (UID: \"bee57d28-4336-4792-aeeb-044ee8338a9b\") " Oct 29 23:08:23.855036 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:23.855025 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bee57d28-4336-4792-aeeb-044ee8338a9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee57d28-4336-4792-aeeb-044ee8338a9b-plugins-config-volume\") pod \"bee57d28-4336-4792-aeeb-044ee8338a9b\" (UID: \"bee57d28-4336-4792-aeeb-044ee8338a9b\") " Oct 29 23:08:23.855445 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:23.855418 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee57d28-4336-4792-aeeb-044ee8338a9b-plugins-config-volume" pod "bee57d28-4336-4792-aeeb-044ee8338a9b" (UID: "bee57d28-4336-4792-aeeb-044ee8338a9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:23.855716 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:23.855586 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bee57d28-4336-4792-aeeb-044ee8338a9b-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:23.855716 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:23.855599 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckpq7\" (UniqueName: \"kubernetes.io/projected/ca53539d-ef76-4a59-8b01-815fe110a113-kube-api-access-ckpq7\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:23.857249 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:23.857149 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee57d28-4336-4792-aeeb-044ee8338a9b-kube-api-access-kf8ml" pod "bee57d28-4336-4792-aeeb-044ee8338a9b" (UID: "bee57d28-4336-4792-aeeb-044ee8338a9b"). InnerVolumeSpecName "kube-api-access-kf8ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:23.887720 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:08:23.887681 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:08:23.956903 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:23.956872 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf8ml\" (UniqueName: \"kubernetes.io/projected/bee57d28-4336-4792-aeeb-044ee8338a9b-kube-api-access-kf8ml\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:24.195550 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:24.195513 4821 generic.go:356] "Generic (PLEG): container finished" podID="bee57d28-4336-4792-aeeb-044ee8338a9b" containerID="b7a212ae818443ffc011b3bb557c5741b863678db7d8d5f3e844357d86e8779c" exitCode=0 Oct 29 23:08:24.195653 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:24.195562 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8" Oct 29 23:08:24.195653 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:24.195585 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8" event={"ID":"bee57d28-4336-4792-aeeb-044ee8338a9b","Type":"ContainerDied","Data":"b7a212ae818443ffc011b3bb557c5741b863678db7d8d5f3e844357d86e8779c"} Oct 29 23:08:24.195732 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:24.195651 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8" event={"ID":"bee57d28-4336-4792-aeeb-044ee8338a9b","Type":"ContainerDied","Data":"069f203d93785598f5d006c6938f2c1c473efac2096ce44ca5589d99e894dcba"} Oct 29 23:08:24.195732 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:24.195672 4821 scope.go:122] "RemoveContainer" containerID="b7a212ae818443ffc011b3bb557c5741b863678db7d8d5f3e844357d86e8779c" Oct 29 23:08:24.197385 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:24.197355 4821 generic.go:356] "Generic (PLEG): container finished" podID="ca53539d-ef76-4a59-8b01-815fe110a113" containerID="fe248149a0611f21bc014106d80ecc0bb6000bc628ff0057852d079b457dc33e" exitCode=0 Oct 29 23:08:24.197562 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:24.197411 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475" Oct 29 23:08:24.197562 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:24.197421 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475" event={"ID":"ca53539d-ef76-4a59-8b01-815fe110a113","Type":"ContainerDied","Data":"fe248149a0611f21bc014106d80ecc0bb6000bc628ff0057852d079b457dc33e"} Oct 29 23:08:24.197562 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:24.197458 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475" event={"ID":"ca53539d-ef76-4a59-8b01-815fe110a113","Type":"ContainerDied","Data":"982afd22f6b0aa17c493aa1292d57c81d5d838492be2218b2fafdab97b8566d8"} Oct 29 23:08:24.207022 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:24.207005 4821 scope.go:122] "RemoveContainer" containerID="b7a212ae818443ffc011b3bb557c5741b863678db7d8d5f3e844357d86e8779c" Oct 29 23:08:24.207339 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:08:24.207316 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7a212ae818443ffc011b3bb557c5741b863678db7d8d5f3e844357d86e8779c\": container with ID starting with b7a212ae818443ffc011b3bb557c5741b863678db7d8d5f3e844357d86e8779c not found: ID does not exist" containerID="b7a212ae818443ffc011b3bb557c5741b863678db7d8d5f3e844357d86e8779c" Oct 29 23:08:24.207422 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:24.207351 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7a212ae818443ffc011b3bb557c5741b863678db7d8d5f3e844357d86e8779c"} err="failed to get container status \"b7a212ae818443ffc011b3bb557c5741b863678db7d8d5f3e844357d86e8779c\": rpc error: code = NotFound desc = could not find container \"b7a212ae818443ffc011b3bb557c5741b863678db7d8d5f3e844357d86e8779c\": container with ID starting with b7a212ae818443ffc011b3bb557c5741b863678db7d8d5f3e844357d86e8779c not found: ID does not exist" Oct 29 23:08:24.207422 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:24.207374 4821 scope.go:122] "RemoveContainer" containerID="fe248149a0611f21bc014106d80ecc0bb6000bc628ff0057852d079b457dc33e" Oct 29 23:08:24.218365 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:24.218343 4821 scope.go:122] "RemoveContainer" containerID="fe248149a0611f21bc014106d80ecc0bb6000bc628ff0057852d079b457dc33e" Oct 29 23:08:24.218659 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:08:24.218639 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe248149a0611f21bc014106d80ecc0bb6000bc628ff0057852d079b457dc33e\": container with ID starting with fe248149a0611f21bc014106d80ecc0bb6000bc628ff0057852d079b457dc33e not found: ID does not exist" containerID="fe248149a0611f21bc014106d80ecc0bb6000bc628ff0057852d079b457dc33e" Oct 29 23:08:24.218723 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:24.218667 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe248149a0611f21bc014106d80ecc0bb6000bc628ff0057852d079b457dc33e"} err="failed to get container status \"fe248149a0611f21bc014106d80ecc0bb6000bc628ff0057852d079b457dc33e\": rpc error: code = NotFound desc = could not find container \"fe248149a0611f21bc014106d80ecc0bb6000bc628ff0057852d079b457dc33e\": container with ID starting with fe248149a0611f21bc014106d80ecc0bb6000bc628ff0057852d079b457dc33e not found: ID does not exist" Oct 29 23:08:24.255895 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:24.255813 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475"] Oct 29 23:08:24.274899 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:24.274866 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-047892-7bdfd8b9c8-kp475"] Oct 29 23:08:24.303480 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:24.303343 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8"] Oct 29 23:08:24.316501 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:24.316468 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-047892-7d9896c7f5-7kbb8"] Oct 29 23:08:25.850409 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:25.850376 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g"] Oct 29 23:08:25.850960 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:25.850643 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g" podUID="562cc122-2430-4052-9085-f67e43a9cbb9" containerName="maas-api" containerID="cri-o://d8b7ed801a9c00ac2a55933540578c0efccd42bf88b9d08ea323ffe8412c421a" gracePeriod=30 Oct 29 23:08:26.176759 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.176715 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bee57d28-4336-4792-aeeb-044ee8338a9b" path="/var/lib/kubelet/pods/bee57d28-4336-4792-aeeb-044ee8338a9b/volumes" Oct 29 23:08:26.177417 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.177394 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca53539d-ef76-4a59-8b01-815fe110a113" path="/var/lib/kubelet/pods/ca53539d-ef76-4a59-8b01-815fe110a113/volumes" Oct 29 23:08:26.182435 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.182409 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv"] Oct 29 23:08:26.183999 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.183973 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-047892-dgnlv"] Oct 29 23:08:26.219903 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.219871 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g" Oct 29 23:08:26.226218 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.226171 4821 generic.go:356] "Generic (PLEG): container finished" podID="562cc122-2430-4052-9085-f67e43a9cbb9" containerID="d8b7ed801a9c00ac2a55933540578c0efccd42bf88b9d08ea323ffe8412c421a" exitCode=0 Oct 29 23:08:26.226362 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.226243 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g" Oct 29 23:08:26.226362 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.226254 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g" event={"ID":"562cc122-2430-4052-9085-f67e43a9cbb9","Type":"ContainerDied","Data":"d8b7ed801a9c00ac2a55933540578c0efccd42bf88b9d08ea323ffe8412c421a"} Oct 29 23:08:26.226362 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.226310 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g" event={"ID":"562cc122-2430-4052-9085-f67e43a9cbb9","Type":"ContainerDied","Data":"33201b517ca8c47eb1d72f59b140ef2c291b40515583f6120602742b0e0a4ab3"} Oct 29 23:08:26.226362 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.226334 4821 scope.go:122] "RemoveContainer" containerID="d8b7ed801a9c00ac2a55933540578c0efccd42bf88b9d08ea323ffe8412c421a" Oct 29 23:08:26.240617 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.240585 4821 scope.go:122] "RemoveContainer" containerID="d8b7ed801a9c00ac2a55933540578c0efccd42bf88b9d08ea323ffe8412c421a" Oct 29 23:08:26.240967 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:08:26.240945 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8b7ed801a9c00ac2a55933540578c0efccd42bf88b9d08ea323ffe8412c421a\": container with ID starting with d8b7ed801a9c00ac2a55933540578c0efccd42bf88b9d08ea323ffe8412c421a not found: ID does not exist" containerID="d8b7ed801a9c00ac2a55933540578c0efccd42bf88b9d08ea323ffe8412c421a" Oct 29 23:08:26.241051 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.240973 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8b7ed801a9c00ac2a55933540578c0efccd42bf88b9d08ea323ffe8412c421a"} err="failed to get container status \"d8b7ed801a9c00ac2a55933540578c0efccd42bf88b9d08ea323ffe8412c421a\": rpc error: code = NotFound desc = could not find container \"d8b7ed801a9c00ac2a55933540578c0efccd42bf88b9d08ea323ffe8412c421a\": container with ID starting with d8b7ed801a9c00ac2a55933540578c0efccd42bf88b9d08ea323ffe8412c421a not found: ID does not exist" Oct 29 23:08:26.461859 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.461706 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/562cc122-2430-4052-9085-f67e43a9cbb9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/562cc122-2430-4052-9085-f67e43a9cbb9-openshift-service-ca\") pod \"562cc122-2430-4052-9085-f67e43a9cbb9\" (UID: \"562cc122-2430-4052-9085-f67e43a9cbb9\") " Oct 29 23:08:26.462052 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.461943 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/562cc122-2430-4052-9085-f67e43a9cbb9-kube-api-access-pl5rh\" (UniqueName: \"kubernetes.io/projected/562cc122-2430-4052-9085-f67e43a9cbb9-kube-api-access-pl5rh\") pod \"562cc122-2430-4052-9085-f67e43a9cbb9\" (UID: \"562cc122-2430-4052-9085-f67e43a9cbb9\") " Oct 29 23:08:26.462121 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.462052 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/562cc122-2430-4052-9085-f67e43a9cbb9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/562cc122-2430-4052-9085-f67e43a9cbb9-metrics-certs\") pod \"562cc122-2430-4052-9085-f67e43a9cbb9\" (UID: \"562cc122-2430-4052-9085-f67e43a9cbb9\") " Oct 29 23:08:26.462187 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.462168 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/562cc122-2430-4052-9085-f67e43a9cbb9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/562cc122-2430-4052-9085-f67e43a9cbb9-maas-api-tls\") pod \"562cc122-2430-4052-9085-f67e43a9cbb9\" (UID: \"562cc122-2430-4052-9085-f67e43a9cbb9\") " Oct 29 23:08:26.535511 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.535472 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/562cc122-2430-4052-9085-f67e43a9cbb9-openshift-service-ca" pod "562cc122-2430-4052-9085-f67e43a9cbb9" (UID: "562cc122-2430-4052-9085-f67e43a9cbb9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:26.535735 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.535552 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/562cc122-2430-4052-9085-f67e43a9cbb9-kube-api-access-pl5rh" pod "562cc122-2430-4052-9085-f67e43a9cbb9" (UID: "562cc122-2430-4052-9085-f67e43a9cbb9"). InnerVolumeSpecName "kube-api-access-pl5rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:26.548809 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.548779 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/562cc122-2430-4052-9085-f67e43a9cbb9-maas-api-tls" pod "562cc122-2430-4052-9085-f67e43a9cbb9" (UID: "562cc122-2430-4052-9085-f67e43a9cbb9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:08:26.555566 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.555541 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/562cc122-2430-4052-9085-f67e43a9cbb9-metrics-certs" pod "562cc122-2430-4052-9085-f67e43a9cbb9" (UID: "562cc122-2430-4052-9085-f67e43a9cbb9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:08:26.614738 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.614540 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/562cc122-2430-4052-9085-f67e43a9cbb9-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:26.614738 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.614596 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/562cc122-2430-4052-9085-f67e43a9cbb9-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:26.614738 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.614614 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl5rh\" (UniqueName: \"kubernetes.io/projected/562cc122-2430-4052-9085-f67e43a9cbb9-kube-api-access-pl5rh\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:26.614738 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.614626 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/562cc122-2430-4052-9085-f67e43a9cbb9-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:26.882679 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.882648 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g"] Oct 29 23:08:26.894065 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.894034 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-77238f-85645c556-kxs5g"] Oct 29 23:08:26.904256 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.900995 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq"] Oct 29 23:08:26.905785 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:26.905762 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-77238f-lh8lq"] Oct 29 23:08:27.180097 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:27.180064 4821 scope.go:122] "RemoveContainer" containerID="6dc55bf1f444b03094c2682fb7e5a033a12d23e245575017542c4be9d529a35e" Oct 29 23:08:27.191528 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:27.191503 4821 scope.go:122] "RemoveContainer" containerID="8f9a4ad0a295170e51f44372908d585faa6efa233e0998901b44703fa6fb63d1" Oct 29 23:08:28.171769 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:28.171735 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07614620-ede6-4242-94e2-57a89c2909f5" path="/var/lib/kubelet/pods/07614620-ede6-4242-94e2-57a89c2909f5/volumes" Oct 29 23:08:28.172420 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:28.172400 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="562cc122-2430-4052-9085-f67e43a9cbb9" path="/var/lib/kubelet/pods/562cc122-2430-4052-9085-f67e43a9cbb9/volumes" Oct 29 23:08:28.172888 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:28.172867 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8000f02e-22df-47a9-ac72-5d245e52218c" path="/var/lib/kubelet/pods/8000f02e-22df-47a9-ac72-5d245e52218c/volumes" Oct 29 23:08:33.710224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:33.698451 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn"] Oct 29 23:08:33.710224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:33.698704 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" podUID="ff6b1260-3dbc-4391-a1a9-7c2de989026c" containerName="istio-proxy" containerID="cri-o://408ae427853f06dc48f45ead4edaba3d9ce7ec4c60a3e9dd6cf15166125b1b16" gracePeriod=30 Oct 29 23:08:33.861237 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:33.851540 4821 patch_prober.go:29] interesting pod/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:08:33.861237 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:33.851615 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" podUID="ff6b1260-3dbc-4391-a1a9-7c2de989026c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:08:34.549618 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:34.549574 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz"] Oct 29 23:08:34.549895 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:34.549843 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" podUID="3adc54c5-a6cf-413f-a347-1397731cf3ff" containerName="istio-proxy" containerID="cri-o://6dbeb918ccc35bc6ae20ac6ff4cad4e2384b04db70a95d70ab4ae51838511fa3" gracePeriod=30 Oct 29 23:08:34.559279 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:34.559241 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk"] Oct 29 23:08:34.559549 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:34.559515 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" podUID="b6671cb5-2dc2-4ced-9aee-33c298a0659e" containerName="istio-proxy" containerID="cri-o://d458af9722a3e4b31d2b7e84871ff6736e0a3fe0af705aaf5fde4beda8dae015" gracePeriod=30 Oct 29 23:08:34.560868 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:34.560841 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4"] Oct 29 23:08:34.561159 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:34.561131 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" podUID="0c0b8707-7668-4b6b-8a0b-4ad8fd956407" containerName="istio-proxy" containerID="cri-o://92143966a1655f2d68309ed9ed4fa0dcb4f1df78d2e7ded7d4ecf2832203c8eb" gracePeriod=30 Oct 29 23:08:36.010858 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:36.010818 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:08:36.011276 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:36.010897 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" podUID="3adc54c5-a6cf-413f-a347-1397731cf3ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:08:36.056148 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:36.056090 4821 patch_prober.go:29] interesting pod/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:08:36.056384 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:36.056146 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" podUID="b6671cb5-2dc2-4ced-9aee-33c298a0659e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:08:38.227842 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:38.227757 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5"] Oct 29 23:08:38.228357 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:38.228014 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" podUID="1e845eec-edd8-4ce2-ad4c-fbfc15fead43" containerName="istio-proxy" containerID="cri-o://b4f66bf9c50adf7317b266ccd161958452cd8fada3efab7ef9697b413d10270c" gracePeriod=30 Oct 29 23:08:39.391870 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.391844 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:08:39.405850 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.405821 4821 generic.go:356] "Generic (PLEG): container finished" podID="ff6b1260-3dbc-4391-a1a9-7c2de989026c" containerID="408ae427853f06dc48f45ead4edaba3d9ce7ec4c60a3e9dd6cf15166125b1b16" exitCode=0 Oct 29 23:08:39.406003 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.405870 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" Oct 29 23:08:39.406003 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.405913 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" event={"ID":"ff6b1260-3dbc-4391-a1a9-7c2de989026c","Type":"ContainerDied","Data":"408ae427853f06dc48f45ead4edaba3d9ce7ec4c60a3e9dd6cf15166125b1b16"} Oct 29 23:08:39.406003 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.405977 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn" event={"ID":"ff6b1260-3dbc-4391-a1a9-7c2de989026c","Type":"ContainerDied","Data":"cb3d2d773e2cf3488bdc65297c23aa746085e3e662400b5457c9021333d4cac2"} Oct 29 23:08:39.406119 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.406006 4821 scope.go:122] "RemoveContainer" containerID="408ae427853f06dc48f45ead4edaba3d9ce7ec4c60a3e9dd6cf15166125b1b16" Oct 29 23:08:39.416422 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.416396 4821 scope.go:122] "RemoveContainer" containerID="59b5413d902abc9442d1e9976f6e4c306a237d1d9b34c974b04adf31e18bbb64" Oct 29 23:08:39.426872 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.426847 4821 scope.go:122] "RemoveContainer" containerID="408ae427853f06dc48f45ead4edaba3d9ce7ec4c60a3e9dd6cf15166125b1b16" Oct 29 23:08:39.427165 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:08:39.427142 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"408ae427853f06dc48f45ead4edaba3d9ce7ec4c60a3e9dd6cf15166125b1b16\": container with ID starting with 408ae427853f06dc48f45ead4edaba3d9ce7ec4c60a3e9dd6cf15166125b1b16 not found: ID does not exist" containerID="408ae427853f06dc48f45ead4edaba3d9ce7ec4c60a3e9dd6cf15166125b1b16" Oct 29 23:08:39.427306 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.427175 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"408ae427853f06dc48f45ead4edaba3d9ce7ec4c60a3e9dd6cf15166125b1b16"} err="failed to get container status \"408ae427853f06dc48f45ead4edaba3d9ce7ec4c60a3e9dd6cf15166125b1b16\": rpc error: code = NotFound desc = could not find container \"408ae427853f06dc48f45ead4edaba3d9ce7ec4c60a3e9dd6cf15166125b1b16\": container with ID starting with 408ae427853f06dc48f45ead4edaba3d9ce7ec4c60a3e9dd6cf15166125b1b16 not found: ID does not exist" Oct 29 23:08:39.427306 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.427192 4821 scope.go:122] "RemoveContainer" containerID="59b5413d902abc9442d1e9976f6e4c306a237d1d9b34c974b04adf31e18bbb64" Oct 29 23:08:39.427536 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:08:39.427513 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59b5413d902abc9442d1e9976f6e4c306a237d1d9b34c974b04adf31e18bbb64\": container with ID starting with 59b5413d902abc9442d1e9976f6e4c306a237d1d9b34c974b04adf31e18bbb64 not found: ID does not exist" containerID="59b5413d902abc9442d1e9976f6e4c306a237d1d9b34c974b04adf31e18bbb64" Oct 29 23:08:39.427644 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.427546 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59b5413d902abc9442d1e9976f6e4c306a237d1d9b34c974b04adf31e18bbb64"} err="failed to get container status \"59b5413d902abc9442d1e9976f6e4c306a237d1d9b34c974b04adf31e18bbb64\": rpc error: code = NotFound desc = could not find container \"59b5413d902abc9442d1e9976f6e4c306a237d1d9b34c974b04adf31e18bbb64\": container with ID starting with 59b5413d902abc9442d1e9976f6e4c306a237d1d9b34c974b04adf31e18bbb64 not found: ID does not exist" Oct 29 23:08:39.466023 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.465990 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-envoy\") pod \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " Oct 29 23:08:39.466239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.466103 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-workload-socket\") pod \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " Oct 29 23:08:39.466239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.466189 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff6b1260-3dbc-4391-a1a9-7c2de989026c-kube-api-access-t2dz6\" (UniqueName: \"kubernetes.io/projected/ff6b1260-3dbc-4391-a1a9-7c2de989026c-kube-api-access-t2dz6\") pod \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " Oct 29 23:08:39.466406 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.466256 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istiod-ca-cert\") pod \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " Oct 29 23:08:39.466406 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.466314 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-token\" (UniqueName: \"kubernetes.io/projected/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-token\") pod \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " Oct 29 23:08:39.466526 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.466473 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-workload-certs\") pod \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " Oct 29 23:08:39.466598 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.466576 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-data\") pod \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " Oct 29 23:08:39.466807 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.466775 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-workload-certs" pod "ff6b1260-3dbc-4391-a1a9-7c2de989026c" (UID: "ff6b1260-3dbc-4391-a1a9-7c2de989026c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:39.466939 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.466907 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-data" pod "ff6b1260-3dbc-4391-a1a9-7c2de989026c" (UID: "ff6b1260-3dbc-4391-a1a9-7c2de989026c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:39.467063 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.466951 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-workload-socket" pod "ff6b1260-3dbc-4391-a1a9-7c2de989026c" (UID: "ff6b1260-3dbc-4391-a1a9-7c2de989026c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:39.467063 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.466955 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-credential-socket\") pod \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " Oct 29 23:08:39.467063 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.467026 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-podinfo\") pod \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\" (UID: \"ff6b1260-3dbc-4391-a1a9-7c2de989026c\") " Oct 29 23:08:39.467302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.466969 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istiod-ca-cert" pod "ff6b1260-3dbc-4391-a1a9-7c2de989026c" (UID: "ff6b1260-3dbc-4391-a1a9-7c2de989026c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:39.467371 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.467314 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-credential-socket" pod "ff6b1260-3dbc-4391-a1a9-7c2de989026c" (UID: "ff6b1260-3dbc-4391-a1a9-7c2de989026c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:39.467891 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.467865 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:39.467994 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.467895 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:39.467994 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.467907 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:39.467994 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.467920 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:39.467994 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.467932 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:39.469484 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.469459 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-envoy" pod "ff6b1260-3dbc-4391-a1a9-7c2de989026c" (UID: "ff6b1260-3dbc-4391-a1a9-7c2de989026c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:39.472908 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.472883 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-podinfo" pod "ff6b1260-3dbc-4391-a1a9-7c2de989026c" (UID: "ff6b1260-3dbc-4391-a1a9-7c2de989026c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:08:39.473122 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.473097 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6b1260-3dbc-4391-a1a9-7c2de989026c-kube-api-access-t2dz6" pod "ff6b1260-3dbc-4391-a1a9-7c2de989026c" (UID: "ff6b1260-3dbc-4391-a1a9-7c2de989026c"). InnerVolumeSpecName "kube-api-access-t2dz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:39.473470 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.473444 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-token" pod "ff6b1260-3dbc-4391-a1a9-7c2de989026c" (UID: "ff6b1260-3dbc-4391-a1a9-7c2de989026c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:39.569516 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.569488 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:39.569516 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.569517 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:39.569733 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.569528 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2dz6\" (UniqueName: \"kubernetes.io/projected/ff6b1260-3dbc-4391-a1a9-7c2de989026c-kube-api-access-t2dz6\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:39.569733 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.569536 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff6b1260-3dbc-4391-a1a9-7c2de989026c-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:39.765019 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.764989 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn"] Oct 29 23:08:39.766820 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.766791 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-047892-openshift-default-d5b59f69-pwzzn"] Oct 29 23:08:39.895956 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.895917 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:08:39.980169 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.979665 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:08:39.980169 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.979677 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istiod-ca-cert\") pod \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " Oct 29 23:08:39.980169 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.979738 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-token\" (UniqueName: \"kubernetes.io/projected/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-token\") pod \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " Oct 29 23:08:39.980169 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.979948 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-kube-api-access-j24gx\" (UniqueName: \"kubernetes.io/projected/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-kube-api-access-j24gx\") pod \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " Oct 29 23:08:39.980169 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.980037 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istiod-ca-cert" pod "0c0b8707-7668-4b6b-8a0b-4ad8fd956407" (UID: "0c0b8707-7668-4b6b-8a0b-4ad8fd956407"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:39.980169 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.980045 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-podinfo\") pod \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " Oct 29 23:08:39.980169 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.980106 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-envoy\") pod \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " Oct 29 23:08:39.980640 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.980245 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-workload-socket\") pod \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " Oct 29 23:08:39.980640 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.980279 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-workload-certs\") pod \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " Oct 29 23:08:39.980640 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.980344 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-credential-socket\") pod \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " Oct 29 23:08:39.980640 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.980375 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-data\") pod \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\" (UID: \"0c0b8707-7668-4b6b-8a0b-4ad8fd956407\") " Oct 29 23:08:39.981185 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.981123 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:39.981501 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.981478 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-data" pod "0c0b8707-7668-4b6b-8a0b-4ad8fd956407" (UID: "0c0b8707-7668-4b6b-8a0b-4ad8fd956407"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:39.982296 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.981516 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-workload-socket" pod "0c0b8707-7668-4b6b-8a0b-4ad8fd956407" (UID: "0c0b8707-7668-4b6b-8a0b-4ad8fd956407"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:39.982296 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.981731 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-workload-certs" pod "0c0b8707-7668-4b6b-8a0b-4ad8fd956407" (UID: "0c0b8707-7668-4b6b-8a0b-4ad8fd956407"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:39.982296 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.981747 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-credential-socket" pod "0c0b8707-7668-4b6b-8a0b-4ad8fd956407" (UID: "0c0b8707-7668-4b6b-8a0b-4ad8fd956407"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:39.985054 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.985014 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-kube-api-access-j24gx" pod "0c0b8707-7668-4b6b-8a0b-4ad8fd956407" (UID: "0c0b8707-7668-4b6b-8a0b-4ad8fd956407"). InnerVolumeSpecName "kube-api-access-j24gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:39.985296 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.985183 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-envoy" pod "0c0b8707-7668-4b6b-8a0b-4ad8fd956407" (UID: "0c0b8707-7668-4b6b-8a0b-4ad8fd956407"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:39.985593 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.985558 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-token" pod "0c0b8707-7668-4b6b-8a0b-4ad8fd956407" (UID: "0c0b8707-7668-4b6b-8a0b-4ad8fd956407"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:39.987483 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.987390 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-podinfo" pod "0c0b8707-7668-4b6b-8a0b-4ad8fd956407" (UID: "0c0b8707-7668-4b6b-8a0b-4ad8fd956407"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:08:39.989072 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:39.989048 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.081650 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-workload-certs\") pod \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.081718 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3adc54c5-a6cf-413f-a347-1397731cf3ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3adc54c5-a6cf-413f-a347-1397731cf3ff-istiod-ca-cert\") pod \"3adc54c5-a6cf-413f-a347-1397731cf3ff\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.081752 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-credential-socket\") pod \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.081805 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-credential-socket\") pod \"3adc54c5-a6cf-413f-a347-1397731cf3ff\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.081854 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3adc54c5-a6cf-413f-a347-1397731cf3ff-kube-api-access-5q5c6\" (UniqueName: \"kubernetes.io/projected/3adc54c5-a6cf-413f-a347-1397731cf3ff-kube-api-access-5q5c6\") pod \"3adc54c5-a6cf-413f-a347-1397731cf3ff\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.081919 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-envoy\") pod \"3adc54c5-a6cf-413f-a347-1397731cf3ff\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.081937 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-token\" (UniqueName: \"kubernetes.io/projected/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-token\") pod \"3adc54c5-a6cf-413f-a347-1397731cf3ff\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.081966 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-workload-socket\") pod \"3adc54c5-a6cf-413f-a347-1397731cf3ff\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.081998 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-podinfo\") pod \"3adc54c5-a6cf-413f-a347-1397731cf3ff\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.082036 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6671cb5-2dc2-4ced-9aee-33c298a0659e-kube-api-access-gcctb\" (UniqueName: \"kubernetes.io/projected/b6671cb5-2dc2-4ced-9aee-33c298a0659e-kube-api-access-gcctb\") pod \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.082058 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-data\") pod \"3adc54c5-a6cf-413f-a347-1397731cf3ff\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.082111 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-workload-certs\") pod \"3adc54c5-a6cf-413f-a347-1397731cf3ff\" (UID: \"3adc54c5-a6cf-413f-a347-1397731cf3ff\") " Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.082165 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-workload-socket\") pod \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.082268 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istiod-ca-cert\") pod \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.082339 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-data\") pod \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.082400 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-envoy\") pod \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.082438 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-token\" (UniqueName: \"kubernetes.io/projected/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-token\") pod \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.082469 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-podinfo\") pod \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\" (UID: \"b6671cb5-2dc2-4ced-9aee-33c298a0659e\") " Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.083229 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j24gx\" (UniqueName: \"kubernetes.io/projected/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-kube-api-access-j24gx\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.083253 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.083267 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.083283 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.083297 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.083312 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.083326 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.083339 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c0b8707-7668-4b6b-8a0b-4ad8fd956407-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.083683 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-credential-socket" pod "3adc54c5-a6cf-413f-a347-1397731cf3ff" (UID: "3adc54c5-a6cf-413f-a347-1397731cf3ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.083940 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3adc54c5-a6cf-413f-a347-1397731cf3ff-istiod-ca-cert" pod "3adc54c5-a6cf-413f-a347-1397731cf3ff" (UID: "3adc54c5-a6cf-413f-a347-1397731cf3ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.084103 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-workload-socket" pod "b6671cb5-2dc2-4ced-9aee-33c298a0659e" (UID: "b6671cb5-2dc2-4ced-9aee-33c298a0659e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.084188 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-data" pod "b6671cb5-2dc2-4ced-9aee-33c298a0659e" (UID: "b6671cb5-2dc2-4ced-9aee-33c298a0659e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.084438 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-data" pod "3adc54c5-a6cf-413f-a347-1397731cf3ff" (UID: "3adc54c5-a6cf-413f-a347-1397731cf3ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.084570 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-workload-certs" pod "b6671cb5-2dc2-4ced-9aee-33c298a0659e" (UID: "b6671cb5-2dc2-4ced-9aee-33c298a0659e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.084605 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istiod-ca-cert" pod "b6671cb5-2dc2-4ced-9aee-33c298a0659e" (UID: "b6671cb5-2dc2-4ced-9aee-33c298a0659e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.084745 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-credential-socket" pod "b6671cb5-2dc2-4ced-9aee-33c298a0659e" (UID: "b6671cb5-2dc2-4ced-9aee-33c298a0659e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:40.085100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.084839 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-workload-socket" pod "3adc54c5-a6cf-413f-a347-1397731cf3ff" (UID: "3adc54c5-a6cf-413f-a347-1397731cf3ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:40.087131 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.085419 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-workload-certs" pod "3adc54c5-a6cf-413f-a347-1397731cf3ff" (UID: "3adc54c5-a6cf-413f-a347-1397731cf3ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:40.087223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.087140 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3adc54c5-a6cf-413f-a347-1397731cf3ff-kube-api-access-5q5c6" pod "3adc54c5-a6cf-413f-a347-1397731cf3ff" (UID: "3adc54c5-a6cf-413f-a347-1397731cf3ff"). InnerVolumeSpecName "kube-api-access-5q5c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:40.087391 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.087322 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-token" pod "3adc54c5-a6cf-413f-a347-1397731cf3ff" (UID: "3adc54c5-a6cf-413f-a347-1397731cf3ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:40.087962 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.087935 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6671cb5-2dc2-4ced-9aee-33c298a0659e-kube-api-access-gcctb" pod "b6671cb5-2dc2-4ced-9aee-33c298a0659e" (UID: "b6671cb5-2dc2-4ced-9aee-33c298a0659e"). InnerVolumeSpecName "kube-api-access-gcctb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:40.088460 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.088435 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-envoy" pod "b6671cb5-2dc2-4ced-9aee-33c298a0659e" (UID: "b6671cb5-2dc2-4ced-9aee-33c298a0659e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:40.089290 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.089169 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-podinfo" pod "3adc54c5-a6cf-413f-a347-1397731cf3ff" (UID: "3adc54c5-a6cf-413f-a347-1397731cf3ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:08:40.089632 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.089611 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-token" pod "b6671cb5-2dc2-4ced-9aee-33c298a0659e" (UID: "b6671cb5-2dc2-4ced-9aee-33c298a0659e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:40.089702 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.089612 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-envoy" pod "3adc54c5-a6cf-413f-a347-1397731cf3ff" (UID: "3adc54c5-a6cf-413f-a347-1397731cf3ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:40.089744 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.089724 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-podinfo" pod "b6671cb5-2dc2-4ced-9aee-33c298a0659e" (UID: "b6671cb5-2dc2-4ced-9aee-33c298a0659e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:08:40.171119 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.171083 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff6b1260-3dbc-4391-a1a9-7c2de989026c" path="/var/lib/kubelet/pods/ff6b1260-3dbc-4391-a1a9-7c2de989026c/volumes" Oct 29 23:08:40.185534 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.185499 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.185668 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.185537 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.185668 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.185554 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3adc54c5-a6cf-413f-a347-1397731cf3ff-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.186029 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.185575 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.186029 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.185864 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.186029 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.185878 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q5c6\" (UniqueName: \"kubernetes.io/projected/3adc54c5-a6cf-413f-a347-1397731cf3ff-kube-api-access-5q5c6\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.186029 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.185892 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.186029 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.185912 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.186029 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.185925 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.186029 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.185939 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.186029 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.185952 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcctb\" (UniqueName: \"kubernetes.io/projected/b6671cb5-2dc2-4ced-9aee-33c298a0659e-kube-api-access-gcctb\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.186029 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.185970 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.186029 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.186005 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3adc54c5-a6cf-413f-a347-1397731cf3ff-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.186434 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.186019 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.186434 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.186056 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.186434 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.186075 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.186434 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.186087 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.186434 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.186099 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6671cb5-2dc2-4ced-9aee-33c298a0659e-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:40.415560 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.415523 4821 generic.go:356] "Generic (PLEG): container finished" podID="0c0b8707-7668-4b6b-8a0b-4ad8fd956407" containerID="92143966a1655f2d68309ed9ed4fa0dcb4f1df78d2e7ded7d4ecf2832203c8eb" exitCode=0 Oct 29 23:08:40.416060 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.415584 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" Oct 29 23:08:40.416060 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.415613 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" event={"ID":"0c0b8707-7668-4b6b-8a0b-4ad8fd956407","Type":"ContainerDied","Data":"92143966a1655f2d68309ed9ed4fa0dcb4f1df78d2e7ded7d4ecf2832203c8eb"} Oct 29 23:08:40.416060 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.415691 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4" event={"ID":"0c0b8707-7668-4b6b-8a0b-4ad8fd956407","Type":"ContainerDied","Data":"e161fdd9c1784de89a86eab6d8e1097f7aae9f85a3b1368471780750bced7670"} Oct 29 23:08:40.416060 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.415718 4821 scope.go:122] "RemoveContainer" containerID="92143966a1655f2d68309ed9ed4fa0dcb4f1df78d2e7ded7d4ecf2832203c8eb" Oct 29 23:08:40.422635 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.422608 4821 generic.go:356] "Generic (PLEG): container finished" podID="3adc54c5-a6cf-413f-a347-1397731cf3ff" containerID="6dbeb918ccc35bc6ae20ac6ff4cad4e2384b04db70a95d70ab4ae51838511fa3" exitCode=0 Oct 29 23:08:40.422897 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.422877 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" event={"ID":"3adc54c5-a6cf-413f-a347-1397731cf3ff","Type":"ContainerDied","Data":"6dbeb918ccc35bc6ae20ac6ff4cad4e2384b04db70a95d70ab4ae51838511fa3"} Oct 29 23:08:40.423038 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.423016 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" event={"ID":"3adc54c5-a6cf-413f-a347-1397731cf3ff","Type":"ContainerDied","Data":"cb3c4e7f8fde960ee7b9db156fb77ce115688797388384f27868248b560e658e"} Oct 29 23:08:40.423280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.423259 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz" Oct 29 23:08:40.427149 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.427122 4821 generic.go:356] "Generic (PLEG): container finished" podID="b6671cb5-2dc2-4ced-9aee-33c298a0659e" containerID="d458af9722a3e4b31d2b7e84871ff6736e0a3fe0af705aaf5fde4beda8dae015" exitCode=0 Oct 29 23:08:40.427303 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.427178 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" Oct 29 23:08:40.427368 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.427172 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" event={"ID":"b6671cb5-2dc2-4ced-9aee-33c298a0659e","Type":"ContainerDied","Data":"d458af9722a3e4b31d2b7e84871ff6736e0a3fe0af705aaf5fde4beda8dae015"} Oct 29 23:08:40.427423 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.427377 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk" event={"ID":"b6671cb5-2dc2-4ced-9aee-33c298a0659e","Type":"ContainerDied","Data":"e0d7d3bd41353768e33b968e89f2f7b4efb29fb626d314c803b24bb1f020a862"} Oct 29 23:08:40.430659 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.430642 4821 scope.go:122] "RemoveContainer" containerID="92143966a1655f2d68309ed9ed4fa0dcb4f1df78d2e7ded7d4ecf2832203c8eb" Oct 29 23:08:40.430916 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:08:40.430896 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92143966a1655f2d68309ed9ed4fa0dcb4f1df78d2e7ded7d4ecf2832203c8eb\": container with ID starting with 92143966a1655f2d68309ed9ed4fa0dcb4f1df78d2e7ded7d4ecf2832203c8eb not found: ID does not exist" containerID="92143966a1655f2d68309ed9ed4fa0dcb4f1df78d2e7ded7d4ecf2832203c8eb" Oct 29 23:08:40.430980 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.430924 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92143966a1655f2d68309ed9ed4fa0dcb4f1df78d2e7ded7d4ecf2832203c8eb"} err="failed to get container status \"92143966a1655f2d68309ed9ed4fa0dcb4f1df78d2e7ded7d4ecf2832203c8eb\": rpc error: code = NotFound desc = could not find container \"92143966a1655f2d68309ed9ed4fa0dcb4f1df78d2e7ded7d4ecf2832203c8eb\": container with ID starting with 92143966a1655f2d68309ed9ed4fa0dcb4f1df78d2e7ded7d4ecf2832203c8eb not found: ID does not exist" Oct 29 23:08:40.430980 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.430937 4821 scope.go:122] "RemoveContainer" containerID="6dbeb918ccc35bc6ae20ac6ff4cad4e2384b04db70a95d70ab4ae51838511fa3" Oct 29 23:08:40.441767 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.441749 4821 scope.go:122] "RemoveContainer" containerID="6dbeb918ccc35bc6ae20ac6ff4cad4e2384b04db70a95d70ab4ae51838511fa3" Oct 29 23:08:40.442046 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:08:40.442022 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dbeb918ccc35bc6ae20ac6ff4cad4e2384b04db70a95d70ab4ae51838511fa3\": container with ID starting with 6dbeb918ccc35bc6ae20ac6ff4cad4e2384b04db70a95d70ab4ae51838511fa3 not found: ID does not exist" containerID="6dbeb918ccc35bc6ae20ac6ff4cad4e2384b04db70a95d70ab4ae51838511fa3" Oct 29 23:08:40.442132 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.442056 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dbeb918ccc35bc6ae20ac6ff4cad4e2384b04db70a95d70ab4ae51838511fa3"} err="failed to get container status \"6dbeb918ccc35bc6ae20ac6ff4cad4e2384b04db70a95d70ab4ae51838511fa3\": rpc error: code = NotFound desc = could not find container \"6dbeb918ccc35bc6ae20ac6ff4cad4e2384b04db70a95d70ab4ae51838511fa3\": container with ID starting with 6dbeb918ccc35bc6ae20ac6ff4cad4e2384b04db70a95d70ab4ae51838511fa3 not found: ID does not exist" Oct 29 23:08:40.442132 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.442079 4821 scope.go:122] "RemoveContainer" containerID="d458af9722a3e4b31d2b7e84871ff6736e0a3fe0af705aaf5fde4beda8dae015" Oct 29 23:08:40.454883 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.454862 4821 scope.go:122] "RemoveContainer" containerID="d458af9722a3e4b31d2b7e84871ff6736e0a3fe0af705aaf5fde4beda8dae015" Oct 29 23:08:40.455568 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:08:40.455544 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d458af9722a3e4b31d2b7e84871ff6736e0a3fe0af705aaf5fde4beda8dae015\": container with ID starting with d458af9722a3e4b31d2b7e84871ff6736e0a3fe0af705aaf5fde4beda8dae015 not found: ID does not exist" containerID="d458af9722a3e4b31d2b7e84871ff6736e0a3fe0af705aaf5fde4beda8dae015" Oct 29 23:08:40.455673 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.455579 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d458af9722a3e4b31d2b7e84871ff6736e0a3fe0af705aaf5fde4beda8dae015"} err="failed to get container status \"d458af9722a3e4b31d2b7e84871ff6736e0a3fe0af705aaf5fde4beda8dae015\": rpc error: code = NotFound desc = could not find container \"d458af9722a3e4b31d2b7e84871ff6736e0a3fe0af705aaf5fde4beda8dae015\": container with ID starting with d458af9722a3e4b31d2b7e84871ff6736e0a3fe0af705aaf5fde4beda8dae015 not found: ID does not exist" Oct 29 23:08:40.474546 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.474522 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk"] Oct 29 23:08:40.485287 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.485264 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-k6cxk"] Oct 29 23:08:40.523877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.523848 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz"] Oct 29 23:08:40.527913 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.527890 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-26cfz"] Oct 29 23:08:40.563478 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.563444 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4"] Oct 29 23:08:40.570440 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.570417 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-77238f-openshift-default-7c64fcf6b7-6f9c4"] Oct 29 23:08:40.920619 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.920585 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b37fde0-mvgkr"] Oct 29 23:08:40.921091 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921078 4821 cpu_manager.go:414] "containerMap: removing container" podUID="bee57d28-4336-4792-aeeb-044ee8338a9b" containerName="payload-pre-processing" Oct 29 23:08:40.921132 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921094 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bee57d28-4336-4792-aeeb-044ee8338a9b" containerName="payload-pre-processing" Oct 29 23:08:40.921132 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921104 4821 cpu_manager.go:414] "containerMap: removing container" podUID="3adc54c5-a6cf-413f-a347-1397731cf3ff" containerName="istio-proxy" Oct 29 23:08:40.921132 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921111 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3adc54c5-a6cf-413f-a347-1397731cf3ff" containerName="istio-proxy" Oct 29 23:08:40.921132 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921126 4821 cpu_manager.go:414] "containerMap: removing container" podUID="ca53539d-ef76-4a59-8b01-815fe110a113" containerName="payload-processing" Oct 29 23:08:40.921132 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921130 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca53539d-ef76-4a59-8b01-815fe110a113" containerName="payload-processing" Oct 29 23:08:40.921375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921137 4821 cpu_manager.go:414] "containerMap: removing container" podUID="562cc122-2430-4052-9085-f67e43a9cbb9" containerName="maas-api" Oct 29 23:08:40.921375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921142 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="562cc122-2430-4052-9085-f67e43a9cbb9" containerName="maas-api" Oct 29 23:08:40.921375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921156 4821 cpu_manager.go:414] "containerMap: removing container" podUID="52620065-ff2f-4acc-b33d-3e283eb5076e" containerName="revoke-keys" Oct 29 23:08:40.921375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921161 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52620065-ff2f-4acc-b33d-3e283eb5076e" containerName="revoke-keys" Oct 29 23:08:40.921375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921167 4821 cpu_manager.go:414] "containerMap: removing container" podUID="ff6b1260-3dbc-4391-a1a9-7c2de989026c" containerName="istio-proxy" Oct 29 23:08:40.921375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921171 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff6b1260-3dbc-4391-a1a9-7c2de989026c" containerName="istio-proxy" Oct 29 23:08:40.921375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921182 4821 cpu_manager.go:414] "containerMap: removing container" podUID="ff6b1260-3dbc-4391-a1a9-7c2de989026c" containerName="istio-proxy" Oct 29 23:08:40.921375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921187 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff6b1260-3dbc-4391-a1a9-7c2de989026c" containerName="istio-proxy" Oct 29 23:08:40.921375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921231 4821 cpu_manager.go:414] "containerMap: removing container" podUID="0c0b8707-7668-4b6b-8a0b-4ad8fd956407" containerName="istio-proxy" Oct 29 23:08:40.921375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921238 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c0b8707-7668-4b6b-8a0b-4ad8fd956407" containerName="istio-proxy" Oct 29 23:08:40.921375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921247 4821 cpu_manager.go:414] "containerMap: removing container" podUID="b6671cb5-2dc2-4ced-9aee-33c298a0659e" containerName="istio-proxy" Oct 29 23:08:40.921375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921254 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6671cb5-2dc2-4ced-9aee-33c298a0659e" containerName="istio-proxy" Oct 29 23:08:40.921375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921360 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="52620065-ff2f-4acc-b33d-3e283eb5076e" containerName="revoke-keys" Oct 29 23:08:40.921375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921370 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff6b1260-3dbc-4391-a1a9-7c2de989026c" containerName="istio-proxy" Oct 29 23:08:40.921375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921376 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff6b1260-3dbc-4391-a1a9-7c2de989026c" containerName="istio-proxy" Oct 29 23:08:40.921818 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921386 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c0b8707-7668-4b6b-8a0b-4ad8fd956407" containerName="istio-proxy" Oct 29 23:08:40.921818 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921394 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6671cb5-2dc2-4ced-9aee-33c298a0659e" containerName="istio-proxy" Oct 29 23:08:40.921818 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921401 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="bee57d28-4336-4792-aeeb-044ee8338a9b" containerName="payload-pre-processing" Oct 29 23:08:40.921818 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921409 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="3adc54c5-a6cf-413f-a347-1397731cf3ff" containerName="istio-proxy" Oct 29 23:08:40.921818 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921419 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca53539d-ef76-4a59-8b01-815fe110a113" containerName="payload-processing" Oct 29 23:08:40.921818 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.921427 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="562cc122-2430-4052-9085-f67e43a9cbb9" containerName="maas-api" Oct 29 23:08:40.926328 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.926307 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b37fde0-mvgkr" Oct 29 23:08:40.942366 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.942319 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:08:40.942366 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.942333 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:08:40.946812 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:40.946790 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b37fde0-mvgkr"] Oct 29 23:08:41.102341 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:41.102250 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpq9n\" (UniqueName: \"kubernetes.io/projected/4adfb991-2490-439e-be0c-58c4ca399704-kube-api-access-zpq9n\") pod \"maas-api-revoke-sub-7b37fde0-mvgkr\" (UID: \"4adfb991-2490-439e-be0c-58c4ca399704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b37fde0-mvgkr" Oct 29 23:08:41.102501 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:41.102397 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4adfb991-2490-439e-be0c-58c4ca399704-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b37fde0-mvgkr\" (UID: \"4adfb991-2490-439e-be0c-58c4ca399704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b37fde0-mvgkr" Oct 29 23:08:41.102543 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:41.102497 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4adfb991-2490-439e-be0c-58c4ca399704-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b37fde0-mvgkr\" (UID: \"4adfb991-2490-439e-be0c-58c4ca399704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b37fde0-mvgkr" Oct 29 23:08:41.204150 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:41.204116 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpq9n\" (UniqueName: \"kubernetes.io/projected/4adfb991-2490-439e-be0c-58c4ca399704-kube-api-access-zpq9n\") pod \"maas-api-revoke-sub-7b37fde0-mvgkr\" (UID: \"4adfb991-2490-439e-be0c-58c4ca399704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b37fde0-mvgkr" Oct 29 23:08:41.204352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:41.204189 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4adfb991-2490-439e-be0c-58c4ca399704-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b37fde0-mvgkr\" (UID: \"4adfb991-2490-439e-be0c-58c4ca399704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b37fde0-mvgkr" Oct 29 23:08:41.204352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:41.204280 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4adfb991-2490-439e-be0c-58c4ca399704-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b37fde0-mvgkr\" (UID: \"4adfb991-2490-439e-be0c-58c4ca399704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b37fde0-mvgkr" Oct 29 23:08:41.204939 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:41.204917 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4adfb991-2490-439e-be0c-58c4ca399704-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b37fde0-mvgkr\" (UID: \"4adfb991-2490-439e-be0c-58c4ca399704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b37fde0-mvgkr" Oct 29 23:08:41.206711 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:41.206685 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4adfb991-2490-439e-be0c-58c4ca399704-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b37fde0-mvgkr\" (UID: \"4adfb991-2490-439e-be0c-58c4ca399704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b37fde0-mvgkr" Oct 29 23:08:41.229016 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:41.228981 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpq9n\" (UniqueName: \"kubernetes.io/projected/4adfb991-2490-439e-be0c-58c4ca399704-kube-api-access-zpq9n\") pod \"maas-api-revoke-sub-7b37fde0-mvgkr\" (UID: \"4adfb991-2490-439e-be0c-58c4ca399704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b37fde0-mvgkr" Oct 29 23:08:41.239858 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:41.239836 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b37fde0-mvgkr" Oct 29 23:08:41.549829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:41.549796 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b37fde0-mvgkr"] Oct 29 23:08:41.550477 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:08:41.550453 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4adfb991_2490_439e_be0c_58c4ca399704.slice/crio-7c507cb2176cd954fb24dc5292d7f47514d28a7d6176e29e9e8e67955d9c23ea WatchSource:0}: Error finding container 7c507cb2176cd954fb24dc5292d7f47514d28a7d6176e29e9e8e67955d9c23ea: Status 404 returned error can't find the container with id 7c507cb2176cd954fb24dc5292d7f47514d28a7d6176e29e9e8e67955d9c23ea Oct 29 23:08:42.171923 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:42.171887 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c0b8707-7668-4b6b-8a0b-4ad8fd956407" path="/var/lib/kubelet/pods/0c0b8707-7668-4b6b-8a0b-4ad8fd956407/volumes" Oct 29 23:08:42.172549 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:42.172525 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3adc54c5-a6cf-413f-a347-1397731cf3ff" path="/var/lib/kubelet/pods/3adc54c5-a6cf-413f-a347-1397731cf3ff/volumes" Oct 29 23:08:42.173055 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:42.173037 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6671cb5-2dc2-4ced-9aee-33c298a0659e" path="/var/lib/kubelet/pods/b6671cb5-2dc2-4ced-9aee-33c298a0659e/volumes" Oct 29 23:08:42.454301 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:42.454190 4821 generic.go:356] "Generic (PLEG): container finished" podID="4adfb991-2490-439e-be0c-58c4ca399704" containerID="1871320a06307a5e8df7fdd9dd5bddc3e9ee3540ae7dc23925b39a3fe06e2c10" exitCode=0 Oct 29 23:08:42.454301 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:42.454231 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b37fde0-mvgkr" event={"ID":"4adfb991-2490-439e-be0c-58c4ca399704","Type":"ContainerDied","Data":"1871320a06307a5e8df7fdd9dd5bddc3e9ee3540ae7dc23925b39a3fe06e2c10"} Oct 29 23:08:42.454301 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:42.454294 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b37fde0-mvgkr" event={"ID":"4adfb991-2490-439e-be0c-58c4ca399704","Type":"ContainerStarted","Data":"7c507cb2176cd954fb24dc5292d7f47514d28a7d6176e29e9e8e67955d9c23ea"} Oct 29 23:08:43.483400 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.483364 4821 generic.go:356] "Generic (PLEG): container finished" podID="1e845eec-edd8-4ce2-ad4c-fbfc15fead43" containerID="b4f66bf9c50adf7317b266ccd161958452cd8fada3efab7ef9697b413d10270c" exitCode=0 Oct 29 23:08:43.483821 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.483437 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" event={"ID":"1e845eec-edd8-4ce2-ad4c-fbfc15fead43","Type":"ContainerDied","Data":"b4f66bf9c50adf7317b266ccd161958452cd8fada3efab7ef9697b413d10270c"} Oct 29 23:08:43.483821 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.483542 4821 scope.go:122] "RemoveContainer" containerID="0a7dc024e82fada74a07789f502359da954a9573b26b37e5715885b2ee9f6fd4" Oct 29 23:08:43.526558 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.526533 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:08:43.640833 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.640803 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-envoy\") pod \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " Oct 29 23:08:43.640966 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.640870 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-credential-socket\") pod \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " Oct 29 23:08:43.641041 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.640972 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-podinfo\") pod \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " Oct 29 23:08:43.641099 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.641041 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istiod-ca-cert\") pod \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " Oct 29 23:08:43.641160 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.641128 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-workload-socket\") pod \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " Oct 29 23:08:43.641242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.641168 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-workload-certs\") pod \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " Oct 29 23:08:43.641242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.641185 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-credential-socket" pod "1e845eec-edd8-4ce2-ad4c-fbfc15fead43" (UID: "1e845eec-edd8-4ce2-ad4c-fbfc15fead43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:43.641362 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.641247 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-kube-api-access-762xw\" (UniqueName: \"kubernetes.io/projected/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-kube-api-access-762xw\") pod \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " Oct 29 23:08:43.641362 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.641284 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-token\" (UniqueName: \"kubernetes.io/projected/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-token\") pod \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " Oct 29 23:08:43.641362 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.641323 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-data\") pod \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\" (UID: \"1e845eec-edd8-4ce2-ad4c-fbfc15fead43\") " Oct 29 23:08:43.641523 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.641384 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-workload-socket" pod "1e845eec-edd8-4ce2-ad4c-fbfc15fead43" (UID: "1e845eec-edd8-4ce2-ad4c-fbfc15fead43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:43.641523 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.641430 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istiod-ca-cert" pod "1e845eec-edd8-4ce2-ad4c-fbfc15fead43" (UID: "1e845eec-edd8-4ce2-ad4c-fbfc15fead43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:43.641635 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.641589 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-workload-certs" pod "1e845eec-edd8-4ce2-ad4c-fbfc15fead43" (UID: "1e845eec-edd8-4ce2-ad4c-fbfc15fead43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:43.641903 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.641878 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:43.642105 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.641911 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:43.642105 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.641932 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:43.642105 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.641948 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:43.642105 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.642059 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-data" pod "1e845eec-edd8-4ce2-ad4c-fbfc15fead43" (UID: "1e845eec-edd8-4ce2-ad4c-fbfc15fead43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:43.643295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.643269 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-envoy" pod "1e845eec-edd8-4ce2-ad4c-fbfc15fead43" (UID: "1e845eec-edd8-4ce2-ad4c-fbfc15fead43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:08:43.643499 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.643472 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-podinfo" pod "1e845eec-edd8-4ce2-ad4c-fbfc15fead43" (UID: "1e845eec-edd8-4ce2-ad4c-fbfc15fead43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:08:43.643768 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.643752 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-token" pod "1e845eec-edd8-4ce2-ad4c-fbfc15fead43" (UID: "1e845eec-edd8-4ce2-ad4c-fbfc15fead43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:43.644017 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.643996 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-kube-api-access-762xw" pod "1e845eec-edd8-4ce2-ad4c-fbfc15fead43" (UID: "1e845eec-edd8-4ce2-ad4c-fbfc15fead43"). InnerVolumeSpecName "kube-api-access-762xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:43.656125 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.656102 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b37fde0-mvgkr" Oct 29 23:08:43.742680 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.742645 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4adfb991-2490-439e-be0c-58c4ca399704-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4adfb991-2490-439e-be0c-58c4ca399704-maas-api-openshift-ca\") pod \"4adfb991-2490-439e-be0c-58c4ca399704\" (UID: \"4adfb991-2490-439e-be0c-58c4ca399704\") " Oct 29 23:08:43.742888 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.742728 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4adfb991-2490-439e-be0c-58c4ca399704-kube-api-access-zpq9n\" (UniqueName: \"kubernetes.io/projected/4adfb991-2490-439e-be0c-58c4ca399704-kube-api-access-zpq9n\") pod \"4adfb991-2490-439e-be0c-58c4ca399704\" (UID: \"4adfb991-2490-439e-be0c-58c4ca399704\") " Oct 29 23:08:43.742888 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.742774 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4adfb991-2490-439e-be0c-58c4ca399704-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4adfb991-2490-439e-be0c-58c4ca399704-maas-api-xks-ca\") pod \"4adfb991-2490-439e-be0c-58c4ca399704\" (UID: \"4adfb991-2490-439e-be0c-58c4ca399704\") " Oct 29 23:08:43.743073 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.743050 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4adfb991-2490-439e-be0c-58c4ca399704-maas-api-openshift-ca" pod "4adfb991-2490-439e-be0c-58c4ca399704" (UID: "4adfb991-2490-439e-be0c-58c4ca399704"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:43.743362 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.743334 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4adfb991-2490-439e-be0c-58c4ca399704-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:43.743362 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.743363 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-762xw\" (UniqueName: \"kubernetes.io/projected/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-kube-api-access-762xw\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:43.743585 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.743379 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:43.743585 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.743394 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:43.743585 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.743407 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:43.743585 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.743420 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e845eec-edd8-4ce2-ad4c-fbfc15fead43-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:43.745111 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.745052 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4adfb991-2490-439e-be0c-58c4ca399704-maas-api-xks-ca" pod "4adfb991-2490-439e-be0c-58c4ca399704" (UID: "4adfb991-2490-439e-be0c-58c4ca399704"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:08:43.745240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.745157 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4adfb991-2490-439e-be0c-58c4ca399704-kube-api-access-zpq9n" pod "4adfb991-2490-439e-be0c-58c4ca399704" (UID: "4adfb991-2490-439e-be0c-58c4ca399704"). InnerVolumeSpecName "kube-api-access-zpq9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:43.844411 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.844373 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4adfb991-2490-439e-be0c-58c4ca399704-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:43.844411 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:43.844405 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpq9n\" (UniqueName: \"kubernetes.io/projected/4adfb991-2490-439e-be0c-58c4ca399704-kube-api-access-zpq9n\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:44.149662 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.149630 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18fab4db-2cwnf"] Oct 29 23:08:44.150163 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.150150 4821 cpu_manager.go:414] "containerMap: removing container" podUID="1e845eec-edd8-4ce2-ad4c-fbfc15fead43" containerName="istio-proxy" Oct 29 23:08:44.150242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.150165 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e845eec-edd8-4ce2-ad4c-fbfc15fead43" containerName="istio-proxy" Oct 29 23:08:44.150242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.150177 4821 cpu_manager.go:414] "containerMap: removing container" podUID="4adfb991-2490-439e-be0c-58c4ca399704" containerName="revoke-keys" Oct 29 23:08:44.150242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.150183 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4adfb991-2490-439e-be0c-58c4ca399704" containerName="revoke-keys" Oct 29 23:08:44.150242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.150189 4821 cpu_manager.go:414] "containerMap: removing container" podUID="1e845eec-edd8-4ce2-ad4c-fbfc15fead43" containerName="istio-proxy" Oct 29 23:08:44.150242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.150194 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e845eec-edd8-4ce2-ad4c-fbfc15fead43" containerName="istio-proxy" Oct 29 23:08:44.150419 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.150291 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e845eec-edd8-4ce2-ad4c-fbfc15fead43" containerName="istio-proxy" Oct 29 23:08:44.150419 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.150302 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="4adfb991-2490-439e-be0c-58c4ca399704" containerName="revoke-keys" Oct 29 23:08:44.150419 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.150308 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e845eec-edd8-4ce2-ad4c-fbfc15fead43" containerName="istio-proxy" Oct 29 23:08:44.155491 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.155457 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fab4db-2cwnf" Oct 29 23:08:44.179794 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.179763 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18fab4db-2cwnf"] Oct 29 23:08:44.250468 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.250431 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bvtr\" (UniqueName: \"kubernetes.io/projected/91de5578-2f94-4e6c-81e5-70529f2f19d1-kube-api-access-2bvtr\") pod \"maas-api-revoke-sub-18fab4db-2cwnf\" (UID: \"91de5578-2f94-4e6c-81e5-70529f2f19d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fab4db-2cwnf" Oct 29 23:08:44.250648 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.250627 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91de5578-2f94-4e6c-81e5-70529f2f19d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18fab4db-2cwnf\" (UID: \"91de5578-2f94-4e6c-81e5-70529f2f19d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fab4db-2cwnf" Oct 29 23:08:44.250756 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.250697 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91de5578-2f94-4e6c-81e5-70529f2f19d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18fab4db-2cwnf\" (UID: \"91de5578-2f94-4e6c-81e5-70529f2f19d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fab4db-2cwnf" Oct 29 23:08:44.351900 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.351865 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bvtr\" (UniqueName: \"kubernetes.io/projected/91de5578-2f94-4e6c-81e5-70529f2f19d1-kube-api-access-2bvtr\") pod \"maas-api-revoke-sub-18fab4db-2cwnf\" (UID: \"91de5578-2f94-4e6c-81e5-70529f2f19d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fab4db-2cwnf" Oct 29 23:08:44.352101 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.351973 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91de5578-2f94-4e6c-81e5-70529f2f19d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18fab4db-2cwnf\" (UID: \"91de5578-2f94-4e6c-81e5-70529f2f19d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fab4db-2cwnf" Oct 29 23:08:44.352101 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.352025 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91de5578-2f94-4e6c-81e5-70529f2f19d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18fab4db-2cwnf\" (UID: \"91de5578-2f94-4e6c-81e5-70529f2f19d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fab4db-2cwnf" Oct 29 23:08:44.352733 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.352691 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91de5578-2f94-4e6c-81e5-70529f2f19d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18fab4db-2cwnf\" (UID: \"91de5578-2f94-4e6c-81e5-70529f2f19d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fab4db-2cwnf" Oct 29 23:08:44.354612 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.354590 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91de5578-2f94-4e6c-81e5-70529f2f19d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18fab4db-2cwnf\" (UID: \"91de5578-2f94-4e6c-81e5-70529f2f19d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fab4db-2cwnf" Oct 29 23:08:44.376504 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.376478 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bvtr\" (UniqueName: \"kubernetes.io/projected/91de5578-2f94-4e6c-81e5-70529f2f19d1-kube-api-access-2bvtr\") pod \"maas-api-revoke-sub-18fab4db-2cwnf\" (UID: \"91de5578-2f94-4e6c-81e5-70529f2f19d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fab4db-2cwnf" Oct 29 23:08:44.471384 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.471301 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fab4db-2cwnf" Oct 29 23:08:44.494164 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.494141 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b37fde0-mvgkr" Oct 29 23:08:44.494569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.494181 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b37fde0-mvgkr" event={"ID":"4adfb991-2490-439e-be0c-58c4ca399704","Type":"ContainerDied","Data":"7c507cb2176cd954fb24dc5292d7f47514d28a7d6176e29e9e8e67955d9c23ea"} Oct 29 23:08:44.494569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.494270 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c507cb2176cd954fb24dc5292d7f47514d28a7d6176e29e9e8e67955d9c23ea" Oct 29 23:08:44.496364 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.496342 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" Oct 29 23:08:44.496482 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.496339 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5" event={"ID":"1e845eec-edd8-4ce2-ad4c-fbfc15fead43","Type":"ContainerDied","Data":"57ee585fd21f061fec1ea4eae02f6f1425ee6df139aba84d242960857ba0c39c"} Oct 29 23:08:44.496536 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.496497 4821 scope.go:122] "RemoveContainer" containerID="b4f66bf9c50adf7317b266ccd161958452cd8fada3efab7ef9697b413d10270c" Oct 29 23:08:44.611640 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.611612 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5"] Oct 29 23:08:44.616170 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.616143 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-d67deb-openshift-default-79c88f87b5-x2gh5"] Oct 29 23:08:44.779688 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:44.779654 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18fab4db-2cwnf"] Oct 29 23:08:44.780449 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:08:44.780422 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91de5578_2f94_4e6c_81e5_70529f2f19d1.slice/crio-f97af94ea08ca1b505078d91a8d9a9f17ea5ba0d1c759699a5d09b271a79d549 WatchSource:0}: Error finding container f97af94ea08ca1b505078d91a8d9a9f17ea5ba0d1c759699a5d09b271a79d549: Status 404 returned error can't find the container with id f97af94ea08ca1b505078d91a8d9a9f17ea5ba0d1c759699a5d09b271a79d549 Oct 29 23:08:45.505696 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:45.505645 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fab4db-2cwnf" event={"ID":"91de5578-2f94-4e6c-81e5-70529f2f19d1","Type":"ContainerStarted","Data":"d3474112506867ad785d0bd631ff6ea3d0ef2083402ddfb6bc9273b7d64b0165"} Oct 29 23:08:45.506328 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:45.505726 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fab4db-2cwnf" event={"ID":"91de5578-2f94-4e6c-81e5-70529f2f19d1","Type":"ContainerStarted","Data":"f97af94ea08ca1b505078d91a8d9a9f17ea5ba0d1c759699a5d09b271a79d549"} Oct 29 23:08:45.551305 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:45.551257 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fab4db-2cwnf" podStartSLOduration=1.551243381 podStartE2EDuration="1.551243381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:45.547244601 +0000 UTC m=+1579.487792512" watchObservedRunningTime="2026-10-29 23:08:45.551243381 +0000 UTC m=+1579.491791306" Oct 29 23:08:46.171382 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:46.171338 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e845eec-edd8-4ce2-ad4c-fbfc15fead43" path="/var/lib/kubelet/pods/1e845eec-edd8-4ce2-ad4c-fbfc15fead43/volumes" Oct 29 23:08:46.515857 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:46.515769 4821 generic.go:356] "Generic (PLEG): container finished" podID="91de5578-2f94-4e6c-81e5-70529f2f19d1" containerID="d3474112506867ad785d0bd631ff6ea3d0ef2083402ddfb6bc9273b7d64b0165" exitCode=0 Oct 29 23:08:46.516285 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:46.515902 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fab4db-2cwnf" event={"ID":"91de5578-2f94-4e6c-81e5-70529f2f19d1","Type":"ContainerDied","Data":"d3474112506867ad785d0bd631ff6ea3d0ef2083402ddfb6bc9273b7d64b0165"} Oct 29 23:08:47.686772 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:47.686744 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fab4db-2cwnf" Oct 29 23:08:47.801312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:47.800355 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91de5578-2f94-4e6c-81e5-70529f2f19d1-kube-api-access-2bvtr\" (UniqueName: \"kubernetes.io/projected/91de5578-2f94-4e6c-81e5-70529f2f19d1-kube-api-access-2bvtr\") pod \"91de5578-2f94-4e6c-81e5-70529f2f19d1\" (UID: \"91de5578-2f94-4e6c-81e5-70529f2f19d1\") " Oct 29 23:08:47.801312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:47.800484 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91de5578-2f94-4e6c-81e5-70529f2f19d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91de5578-2f94-4e6c-81e5-70529f2f19d1-maas-api-xks-ca\") pod \"91de5578-2f94-4e6c-81e5-70529f2f19d1\" (UID: \"91de5578-2f94-4e6c-81e5-70529f2f19d1\") " Oct 29 23:08:47.801312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:47.800595 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91de5578-2f94-4e6c-81e5-70529f2f19d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91de5578-2f94-4e6c-81e5-70529f2f19d1-maas-api-openshift-ca\") pod \"91de5578-2f94-4e6c-81e5-70529f2f19d1\" (UID: \"91de5578-2f94-4e6c-81e5-70529f2f19d1\") " Oct 29 23:08:47.801312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:47.801229 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91de5578-2f94-4e6c-81e5-70529f2f19d1-maas-api-openshift-ca" pod "91de5578-2f94-4e6c-81e5-70529f2f19d1" (UID: "91de5578-2f94-4e6c-81e5-70529f2f19d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:47.805478 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:47.803659 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91de5578-2f94-4e6c-81e5-70529f2f19d1-maas-api-xks-ca" pod "91de5578-2f94-4e6c-81e5-70529f2f19d1" (UID: "91de5578-2f94-4e6c-81e5-70529f2f19d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:08:47.805478 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:47.804095 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91de5578-2f94-4e6c-81e5-70529f2f19d1-kube-api-access-2bvtr" pod "91de5578-2f94-4e6c-81e5-70529f2f19d1" (UID: "91de5578-2f94-4e6c-81e5-70529f2f19d1"). InnerVolumeSpecName "kube-api-access-2bvtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:47.902676 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:47.902642 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bvtr\" (UniqueName: \"kubernetes.io/projected/91de5578-2f94-4e6c-81e5-70529f2f19d1-kube-api-access-2bvtr\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:47.902676 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:47.902672 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91de5578-2f94-4e6c-81e5-70529f2f19d1-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:47.902676 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:47.902684 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91de5578-2f94-4e6c-81e5-70529f2f19d1-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:48.533987 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:48.533896 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fab4db-2cwnf" event={"ID":"91de5578-2f94-4e6c-81e5-70529f2f19d1","Type":"ContainerDied","Data":"f97af94ea08ca1b505078d91a8d9a9f17ea5ba0d1c759699a5d09b271a79d549"} Oct 29 23:08:48.533987 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:48.533923 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fab4db-2cwnf" Oct 29 23:08:48.533987 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:48.533946 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f97af94ea08ca1b505078d91a8d9a9f17ea5ba0d1c759699a5d09b271a79d549" Oct 29 23:08:52.258406 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:52.258356 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:52.313389 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:52.313344 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 29 23:08:52.416344 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:52.416294 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:52.426361 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:52.426319 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 23:08:52.446335 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:52.446289 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:52.484331 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:52.484288 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 29 23:08:52.660895 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:52.659847 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:52.660895 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:52.660047 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:52.673229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:52.671972 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:52.673229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:52.672312 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 29 23:08:52.673229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:52.672613 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:52.700162 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:52.699105 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:52.915284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:52.914429 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:53.004568 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.004512 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:08:53.004890 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.004843 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" podUID="ae79f378-d055-4e08-bd3a-1e75fcc38882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:53.260287 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.259350 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:53.260287 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.260060 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 23:08:53.364662 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.364613 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:53.365184 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.365149 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 23:08:53.452416 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.452355 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:53.487235 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.479533 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 23:08:53.545284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.529224 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:53.545861 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.529364 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 29 23:08:53.575228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.546668 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 23:08:53.662385 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.562444 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 29 23:08:53.662385 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.636746 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 23:08:53.662385 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.636801 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 23:08:53.666335 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.664240 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:53.666335 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.664580 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 29 23:08:53.666335 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.665136 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 23:08:53.666335 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.665188 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 23:08:53.694302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.668276 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:53.694302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.678322 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:53.694302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.678733 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 29 23:08:53.694302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.679034 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 23:08:53.694302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.679101 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 23:08:53.697383 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.697348 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:08:53.697652 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.697604 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" podUID="23d4c06e-ed49-45da-85d5-55e60a3d48c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:53.705152 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.705118 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:53.705727 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.705703 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 23:08:53.806282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.803315 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 23:08:53.828520 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.825986 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded" Oct 29 23:08:53.877304 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.877253 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:53.928228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.884130 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 23:08:53.928228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.912313 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 23:08:53.965664 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.963259 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 23:08:53.985258 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.969859 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 23:08:53.985258 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:53.979543 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 23:08:54.064907 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:54.064813 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 23:08:54.068326 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:54.066713 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 23:08:54.068326 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:54.066976 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 23:08:54.072263 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:54.070162 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 23:08:54.073682 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:54.073615 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 23:08:54.078006 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:08:54.077979 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:08:55.622798 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:55.622763 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3bfcdda-dlkrg"] Oct 29 23:08:55.623541 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:55.623522 4821 cpu_manager.go:414] "containerMap: removing container" podUID="91de5578-2f94-4e6c-81e5-70529f2f19d1" containerName="revoke-keys" Oct 29 23:08:55.623658 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:55.623544 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91de5578-2f94-4e6c-81e5-70529f2f19d1" containerName="revoke-keys" Oct 29 23:08:55.623739 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:55.623726 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="91de5578-2f94-4e6c-81e5-70529f2f19d1" containerName="revoke-keys" Oct 29 23:08:55.627521 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:55.627500 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bfcdda-dlkrg" Oct 29 23:08:55.654912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:55.654878 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3bfcdda-dlkrg"] Oct 29 23:08:55.682900 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:55.682865 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:08:55.683051 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:55.682906 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:08:55.686633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:55.686599 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13abc4a1-190c-49b8-9603-6f64c96f9f1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3bfcdda-dlkrg\" (UID: \"13abc4a1-190c-49b8-9603-6f64c96f9f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bfcdda-dlkrg" Oct 29 23:08:55.686818 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:55.686784 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvchv\" (UniqueName: \"kubernetes.io/projected/13abc4a1-190c-49b8-9603-6f64c96f9f1c-kube-api-access-nvchv\") pod \"maas-api-revoke-sub-a3bfcdda-dlkrg\" (UID: \"13abc4a1-190c-49b8-9603-6f64c96f9f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bfcdda-dlkrg" Oct 29 23:08:55.686965 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:55.686946 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13abc4a1-190c-49b8-9603-6f64c96f9f1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3bfcdda-dlkrg\" (UID: \"13abc4a1-190c-49b8-9603-6f64c96f9f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bfcdda-dlkrg" Oct 29 23:08:55.788475 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:55.788433 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvchv\" (UniqueName: \"kubernetes.io/projected/13abc4a1-190c-49b8-9603-6f64c96f9f1c-kube-api-access-nvchv\") pod \"maas-api-revoke-sub-a3bfcdda-dlkrg\" (UID: \"13abc4a1-190c-49b8-9603-6f64c96f9f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bfcdda-dlkrg" Oct 29 23:08:55.788667 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:55.788555 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13abc4a1-190c-49b8-9603-6f64c96f9f1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3bfcdda-dlkrg\" (UID: \"13abc4a1-190c-49b8-9603-6f64c96f9f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bfcdda-dlkrg" Oct 29 23:08:55.788913 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:55.788719 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13abc4a1-190c-49b8-9603-6f64c96f9f1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3bfcdda-dlkrg\" (UID: \"13abc4a1-190c-49b8-9603-6f64c96f9f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bfcdda-dlkrg" Oct 29 23:08:55.789513 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:55.789485 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13abc4a1-190c-49b8-9603-6f64c96f9f1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3bfcdda-dlkrg\" (UID: \"13abc4a1-190c-49b8-9603-6f64c96f9f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bfcdda-dlkrg" Oct 29 23:08:55.791506 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:55.791482 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13abc4a1-190c-49b8-9603-6f64c96f9f1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3bfcdda-dlkrg\" (UID: \"13abc4a1-190c-49b8-9603-6f64c96f9f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bfcdda-dlkrg" Oct 29 23:08:55.824226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:55.814772 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvchv\" (UniqueName: \"kubernetes.io/projected/13abc4a1-190c-49b8-9603-6f64c96f9f1c-kube-api-access-nvchv\") pod \"maas-api-revoke-sub-a3bfcdda-dlkrg\" (UID: \"13abc4a1-190c-49b8-9603-6f64c96f9f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bfcdda-dlkrg" Oct 29 23:08:55.942587 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:55.942469 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bfcdda-dlkrg" Oct 29 23:08:56.260344 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:08:56.260313 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13abc4a1_190c_49b8_9603_6f64c96f9f1c.slice/crio-57ae7594a1c4e1ba7de1d700dc7d7567ce4ab0661bb72d45fb432dee6f9b4687 WatchSource:0}: Error finding container 57ae7594a1c4e1ba7de1d700dc7d7567ce4ab0661bb72d45fb432dee6f9b4687: Status 404 returned error can't find the container with id 57ae7594a1c4e1ba7de1d700dc7d7567ce4ab0661bb72d45fb432dee6f9b4687 Oct 29 23:08:56.260698 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:56.260666 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3bfcdda-dlkrg"] Oct 29 23:08:56.775911 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:56.775809 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bfcdda-dlkrg" event={"ID":"13abc4a1-190c-49b8-9603-6f64c96f9f1c","Type":"ContainerStarted","Data":"a215e3d0cde79944ffa5357b90a0c81a494071d8d60fa9b54f3fb237b08411dc"} Oct 29 23:08:56.775911 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:56.775882 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bfcdda-dlkrg" event={"ID":"13abc4a1-190c-49b8-9603-6f64c96f9f1c","Type":"ContainerStarted","Data":"57ae7594a1c4e1ba7de1d700dc7d7567ce4ab0661bb72d45fb432dee6f9b4687"} Oct 29 23:08:56.820675 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:56.820614 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bfcdda-dlkrg" podStartSLOduration=1.8206001299999999 podStartE2EDuration="1.82060013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:56.819313184 +0000 UTC m=+1590.759861086" watchObservedRunningTime="2026-10-29 23:08:56.82060013 +0000 UTC m=+1590.761148033" Oct 29 23:08:57.786275 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:57.786237 4821 generic.go:356] "Generic (PLEG): container finished" podID="13abc4a1-190c-49b8-9603-6f64c96f9f1c" containerID="a215e3d0cde79944ffa5357b90a0c81a494071d8d60fa9b54f3fb237b08411dc" exitCode=0 Oct 29 23:08:57.786790 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:57.786375 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bfcdda-dlkrg" event={"ID":"13abc4a1-190c-49b8-9603-6f64c96f9f1c","Type":"ContainerDied","Data":"a215e3d0cde79944ffa5357b90a0c81a494071d8d60fa9b54f3fb237b08411dc"} Oct 29 23:08:58.965131 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:58.965109 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bfcdda-dlkrg" Oct 29 23:08:59.035759 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:59.035728 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13abc4a1-190c-49b8-9603-6f64c96f9f1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13abc4a1-190c-49b8-9603-6f64c96f9f1c-maas-api-openshift-ca\") pod \"13abc4a1-190c-49b8-9603-6f64c96f9f1c\" (UID: \"13abc4a1-190c-49b8-9603-6f64c96f9f1c\") " Oct 29 23:08:59.035975 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:59.035911 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13abc4a1-190c-49b8-9603-6f64c96f9f1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13abc4a1-190c-49b8-9603-6f64c96f9f1c-maas-api-xks-ca\") pod \"13abc4a1-190c-49b8-9603-6f64c96f9f1c\" (UID: \"13abc4a1-190c-49b8-9603-6f64c96f9f1c\") " Oct 29 23:08:59.035975 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:59.035958 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13abc4a1-190c-49b8-9603-6f64c96f9f1c-kube-api-access-nvchv\" (UniqueName: \"kubernetes.io/projected/13abc4a1-190c-49b8-9603-6f64c96f9f1c-kube-api-access-nvchv\") pod \"13abc4a1-190c-49b8-9603-6f64c96f9f1c\" (UID: \"13abc4a1-190c-49b8-9603-6f64c96f9f1c\") " Oct 29 23:08:59.036123 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:59.036092 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13abc4a1-190c-49b8-9603-6f64c96f9f1c-maas-api-openshift-ca" pod "13abc4a1-190c-49b8-9603-6f64c96f9f1c" (UID: "13abc4a1-190c-49b8-9603-6f64c96f9f1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:08:59.036553 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:59.036529 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13abc4a1-190c-49b8-9603-6f64c96f9f1c-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:59.038321 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:59.038297 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13abc4a1-190c-49b8-9603-6f64c96f9f1c-maas-api-xks-ca" pod "13abc4a1-190c-49b8-9603-6f64c96f9f1c" (UID: "13abc4a1-190c-49b8-9603-6f64c96f9f1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:08:59.038415 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:59.038325 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13abc4a1-190c-49b8-9603-6f64c96f9f1c-kube-api-access-nvchv" pod "13abc4a1-190c-49b8-9603-6f64c96f9f1c" (UID: "13abc4a1-190c-49b8-9603-6f64c96f9f1c"). InnerVolumeSpecName "kube-api-access-nvchv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:59.137937 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:59.137902 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13abc4a1-190c-49b8-9603-6f64c96f9f1c-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:59.137937 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:59.137932 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvchv\" (UniqueName: \"kubernetes.io/projected/13abc4a1-190c-49b8-9603-6f64c96f9f1c-kube-api-access-nvchv\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:08:59.805766 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:59.805735 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bfcdda-dlkrg" Oct 29 23:08:59.805979 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:59.805779 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bfcdda-dlkrg" event={"ID":"13abc4a1-190c-49b8-9603-6f64c96f9f1c","Type":"ContainerDied","Data":"57ae7594a1c4e1ba7de1d700dc7d7567ce4ab0661bb72d45fb432dee6f9b4687"} Oct 29 23:08:59.805979 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:08:59.805837 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57ae7594a1c4e1ba7de1d700dc7d7567ce4ab0661bb72d45fb432dee6f9b4687" Oct 29 23:09:23.650478 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.650436 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk"] Oct 29 23:09:23.650972 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.650957 4821 cpu_manager.go:414] "containerMap: removing container" podUID="13abc4a1-190c-49b8-9603-6f64c96f9f1c" containerName="revoke-keys" Oct 29 23:09:23.651017 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.650973 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13abc4a1-190c-49b8-9603-6f64c96f9f1c" containerName="revoke-keys" Oct 29 23:09:23.651094 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.651083 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="13abc4a1-190c-49b8-9603-6f64c96f9f1c" containerName="revoke-keys" Oct 29 23:09:23.654905 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.654873 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.694919 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.694885 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk"] Oct 29 23:09:23.768517 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.768482 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.768718 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.768563 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.768718 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.768686 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.768827 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.768728 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vgxq\" (UniqueName: \"kubernetes.io/projected/2f71bdab-b1a2-4513-a7f1-2455e4222efa-kube-api-access-2vgxq\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.768827 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.768753 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.768905 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.768848 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.768905 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.768881 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.769016 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.768967 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.769080 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.769059 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.870949 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.870897 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.871152 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.870986 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vgxq\" (UniqueName: \"kubernetes.io/projected/2f71bdab-b1a2-4513-a7f1-2455e4222efa-kube-api-access-2vgxq\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.871152 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.871029 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.871152 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.871141 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.871347 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.871226 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.871347 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.871318 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.871466 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.871353 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.871466 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.871402 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.871579 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.871544 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.871951 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.871925 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-data\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.872631 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.872604 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istiod-ca-cert\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.873289 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.873266 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-workload-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.873710 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.873678 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-workload-certs\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.874895 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.874621 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-envoy\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.874895 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.874854 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-credential-socket\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.876157 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.876127 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-podinfo\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.893753 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:09:23.893715 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:09:23.895987 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.895956 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vgxq\" (UniqueName: \"kubernetes.io/projected/2f71bdab-b1a2-4513-a7f1-2455e4222efa-kube-api-access-2vgxq\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.896132 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.896084 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-token\") pod \"e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:23.981905 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:23.981819 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:24.382713 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:24.371169 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk"] Oct 29 23:09:24.382713 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:09:24.373280 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f71bdab_b1a2_4513_a7f1_2455e4222efa.slice/crio-11f05b804161ef06fc0b6e3f061bb8bcfae92710818edf840af7a99f24bb124d WatchSource:0}: Error finding container 11f05b804161ef06fc0b6e3f061bb8bcfae92710818edf840af7a99f24bb124d: Status 404 returned error can't find the container with id 11f05b804161ef06fc0b6e3f061bb8bcfae92710818edf840af7a99f24bb124d Oct 29 23:09:24.382713 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:24.376688 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:09:24.382713 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:24.376773 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:09:24.382713 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:24.376819 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:09:25.119349 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.119310 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" event={"ID":"2f71bdab-b1a2-4513-a7f1-2455e4222efa","Type":"ContainerStarted","Data":"b2306d76e7c3490ed16fb67412a842b2a467b0b846a0334bc06bb0bcd4835090"} Oct 29 23:09:25.119996 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.119973 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" event={"ID":"2f71bdab-b1a2-4513-a7f1-2455e4222efa","Type":"ContainerStarted","Data":"11f05b804161ef06fc0b6e3f061bb8bcfae92710818edf840af7a99f24bb124d"} Oct 29 23:09:25.243794 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.243728 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" podStartSLOduration=2.243707331 podStartE2EDuration="2.243707331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:25.243247443 +0000 UTC m=+1619.183795371" watchObservedRunningTime="2026-10-29 23:09:25.243707331 +0000 UTC m=+1619.184255241" Oct 29 23:09:25.276860 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.272385 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk"] Oct 29 23:09:25.284181 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.280477 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.291309 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.291267 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd"] Oct 29 23:09:25.323550 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.323517 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.324337 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.323399 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk"] Oct 29 23:09:25.339558 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.339534 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd"] Oct 29 23:09:25.360636 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.360604 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.360952 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.360925 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.361119 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.361103 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.361264 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.361245 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gxdc\" (UniqueName: \"kubernetes.io/projected/2cdd98c2-c079-4880-9835-36d2c4a86de8-kube-api-access-8gxdc\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.361491 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.361473 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.361645 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.361627 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.361781 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.361765 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.361888 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.361874 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.362091 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.362074 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdd98c2-c079-4880-9835-36d2c4a86de8-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.480244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.471153 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.480244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.471230 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.480244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.471276 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.480244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.471303 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gxdc\" (UniqueName: \"kubernetes.io/projected/2cdd98c2-c079-4880-9835-36d2c4a86de8-kube-api-access-8gxdc\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.480244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.471393 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.480244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.471436 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfvlk\" (UniqueName: \"kubernetes.io/projected/be674d35-b876-4f7d-9ceb-8cd285a3f2db-kube-api-access-nfvlk\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.480244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.471469 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.480244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.471507 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.480244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.471550 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.480244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.471581 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.480244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.471643 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.480244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.471713 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.480244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.471761 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.480244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.471801 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdd98c2-c079-4880-9835-36d2c4a86de8-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.480244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.471836 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.480244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.471899 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.480244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.471944 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.480244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.472037 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.530227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.530173 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdd98c2-c079-4880-9835-36d2c4a86de8-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.537478 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.534688 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.537478 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.534960 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.537478 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.535278 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.537478 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.535582 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.842295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.647980 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gxdc\" (UniqueName: \"kubernetes.io/projected/2cdd98c2-c079-4880-9835-36d2c4a86de8-kube-api-access-8gxdc\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.842295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.649058 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.842295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.649116 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.842295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.649162 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.842295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.649258 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.842295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.649309 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.842295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.649367 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.842295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.649466 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.842295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.649507 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfvlk\" (UniqueName: \"kubernetes.io/projected/be674d35-b876-4f7d-9ceb-8cd285a3f2db-kube-api-access-nfvlk\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.842295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.649598 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.842295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.649821 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-workload-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.842295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.650082 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-workload-certs\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.842295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.650363 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-data\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.842295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.650367 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istiod-ca-cert\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.842295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.650610 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-credential-socket\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.842295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.664541 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:09:25.842295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.670707 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.854227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.851773 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.872410 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.872365 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:09:25.888435 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.887286 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:09:25.914363 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.914314 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:09:25.932260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.923980 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:09:25.945224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.941648 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-token\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.948536 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.948506 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-envoy\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:25.994942 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.956750 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:25.994942 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.993505 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:09:25.994942 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.993788 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 29 23:09:25.994942 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.994425 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:25.996163 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:25.996100 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfvlk\" (UniqueName: \"kubernetes.io/projected/be674d35-b876-4f7d-9ceb-8cd285a3f2db-kube-api-access-nfvlk\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:26.012367 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:26.010332 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 29 23:09:26.012367 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:26.010392 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" podUID="2f71bdab-b1a2-4513-a7f1-2455e4222efa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 29 23:09:26.074228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:26.073805 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-podinfo\") pod \"e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:26.078224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:26.076646 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:09:26.086528 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:26.084426 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:09:26.086528 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:26.086268 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 29 23:09:26.098736 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:26.095302 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:09:26.328228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:26.279433 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:26.402230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:26.390266 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:09:26.481188 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:26.481157 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:26.721291 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:26.718288 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 23:09:26.721291 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:26.718929 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 23:09:26.857385 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:26.856297 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:09:26.873912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:26.873305 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 23:09:27.022498 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.022457 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 29 23:09:27.024896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.024859 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 23:09:27.025124 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.025112 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 23:09:27.030721 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.030692 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:09:27.037499 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.031112 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:09:27.040835 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.031342 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 29 23:09:27.041416 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.031564 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 29 23:09:27.042505 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.033857 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 29 23:09:27.042706 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.042658 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" podUID="2f71bdab-b1a2-4513-a7f1-2455e4222efa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 29 23:09:27.088056 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.088014 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 29 23:09:27.088682 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.088651 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:09:27.092257 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.040721 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 23:09:27.092257 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.091689 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 23:09:27.092257 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.091760 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 23:09:27.092257 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.091823 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 23:09:27.092257 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.091884 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 23:09:27.092257 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.091941 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 23:09:27.097942 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.097913 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 29 23:09:27.098825 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.098799 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:09:27.099585 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.099559 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 29 23:09:27.099707 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.099685 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded" Oct 29 23:09:27.099957 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.099940 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 23:09:27.100794 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.100778 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 23:09:27.206865 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.206755 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 23:09:27.294331 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.294220 4821 scope.go:122] "RemoveContainer" containerID="2593eb33b190e716633ab15496c8f44429c57f5517f453a816423e20831d7abd" Oct 29 23:09:27.378671 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.378636 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 23:09:27.441375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.405340 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:09:27.452712 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.452672 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 23:09:27.512441 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.512322 4821 scope.go:122] "RemoveContainer" containerID="9a7cc7639e7e018b362d6fd0d7c2eece496cb980dd1467a8d7d8fe98b062854a" Oct 29 23:09:27.722453 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.702141 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 23:09:27.733521 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.730621 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:09:27.751053 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.751006 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:09:27.751400 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.751372 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:09:27.751564 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.751545 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:09:27.782913 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.782790 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:09:27.818320 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.814243 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk"] Oct 29 23:09:27.840947 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.840902 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:09:27.841274 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.841246 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:09:27.841448 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.841430 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:09:27.864226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.856798 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 23:09:27.864226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.859577 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 23:09:27.923385 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.920311 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 23:09:27.942070 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.933311 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd"] Oct 29 23:09:27.956792 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.956762 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 23:09:27.983918 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.983884 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 29 23:09:27.991352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.991313 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" podUID="2f71bdab-b1a2-4513-a7f1-2455e4222efa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 29 23:09:27.998912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:27.998881 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 23:09:28.027574 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.027534 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:09:28.077222 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.077125 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f36035d-zft54"] Oct 29 23:09:28.091685 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.091637 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:09:28.495272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.113700 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 29 23:09:28.495272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.113910 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 29 23:09:28.495272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.113950 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:09:28.495272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.373405 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f36035d-zft54" Oct 29 23:09:28.496012 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.482797 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d413f3-e64a-4367-948a-5d96ff9775de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f36035d-zft54\" (UID: \"d7d413f3-e64a-4367-948a-5d96ff9775de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f36035d-zft54" Oct 29 23:09:28.496012 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.495406 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9h8k\" (UniqueName: \"kubernetes.io/projected/d7d413f3-e64a-4367-948a-5d96ff9775de-kube-api-access-p9h8k\") pod \"maas-api-revoke-sub-0f36035d-zft54\" (UID: \"d7d413f3-e64a-4367-948a-5d96ff9775de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f36035d-zft54" Oct 29 23:09:28.496012 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.495630 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d413f3-e64a-4367-948a-5d96ff9775de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f36035d-zft54\" (UID: \"d7d413f3-e64a-4367-948a-5d96ff9775de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f36035d-zft54" Oct 29 23:09:28.606186 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.579571 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:09:28.606186 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.579650 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:09:28.606186 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.597410 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d413f3-e64a-4367-948a-5d96ff9775de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f36035d-zft54\" (UID: \"d7d413f3-e64a-4367-948a-5d96ff9775de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f36035d-zft54" Oct 29 23:09:28.606186 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.597508 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9h8k\" (UniqueName: \"kubernetes.io/projected/d7d413f3-e64a-4367-948a-5d96ff9775de-kube-api-access-p9h8k\") pod \"maas-api-revoke-sub-0f36035d-zft54\" (UID: \"d7d413f3-e64a-4367-948a-5d96ff9775de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f36035d-zft54" Oct 29 23:09:28.606186 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.597672 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d413f3-e64a-4367-948a-5d96ff9775de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f36035d-zft54\" (UID: \"d7d413f3-e64a-4367-948a-5d96ff9775de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f36035d-zft54" Oct 29 23:09:28.606186 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.598715 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d413f3-e64a-4367-948a-5d96ff9775de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f36035d-zft54\" (UID: \"d7d413f3-e64a-4367-948a-5d96ff9775de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f36035d-zft54" Oct 29 23:09:28.606186 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.601884 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d413f3-e64a-4367-948a-5d96ff9775de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f36035d-zft54\" (UID: \"d7d413f3-e64a-4367-948a-5d96ff9775de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f36035d-zft54" Oct 29 23:09:28.682661 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.677539 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9h8k\" (UniqueName: \"kubernetes.io/projected/d7d413f3-e64a-4367-948a-5d96ff9775de-kube-api-access-p9h8k\") pod \"maas-api-revoke-sub-0f36035d-zft54\" (UID: \"d7d413f3-e64a-4367-948a-5d96ff9775de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f36035d-zft54" Oct 29 23:09:28.729054 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.729016 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f36035d-zft54"] Oct 29 23:09:28.729369 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.729352 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 23:09:28.729548 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.729536 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 23:09:28.770566 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.770465 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 23:09:28.770566 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.770512 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 23:09:28.770679 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.770572 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 23:09:28.770679 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.770322 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f36035d-zft54" Oct 29 23:09:28.770679 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.770622 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 23:09:28.770679 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.770660 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 23:09:28.770809 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.770714 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 23:09:28.773628 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.773570 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" event={"ID":"be674d35-b876-4f7d-9ceb-8cd285a3f2db","Type":"ContainerStarted","Data":"4581c30751f7b7a4d1c855ea037472ba0a5591ca2d9242d8190c75bf96b645ef"} Oct 29 23:09:28.779534 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:28.779450 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" event={"ID":"2cdd98c2-c079-4880-9835-36d2c4a86de8","Type":"ContainerStarted","Data":"df6cc09625ce9a4bff8800575225fcc9c26c0fac60f0821bdae35d6869ae3012"} Oct 29 23:09:29.001500 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:29.001467 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 29 23:09:29.002539 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:29.001737 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" podUID="2f71bdab-b1a2-4513-a7f1-2455e4222efa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 29 23:09:29.274822 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:29.274798 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f36035d-zft54"] Oct 29 23:09:29.797697 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:29.797657 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" event={"ID":"2cdd98c2-c079-4880-9835-36d2c4a86de8","Type":"ContainerStarted","Data":"c9aae7a6235e709152958c1867747e4fa2f852bd6d7f2a2dabfc43b67fa20e64"} Oct 29 23:09:29.799614 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:29.799585 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f36035d-zft54" event={"ID":"d7d413f3-e64a-4367-948a-5d96ff9775de","Type":"ContainerStarted","Data":"0de46000b6e40696ec110dee879176c9e192ccf14b1fcf07698f28e5e6ef0e52"} Oct 29 23:09:29.801774 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:29.801738 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" event={"ID":"be674d35-b876-4f7d-9ceb-8cd285a3f2db","Type":"ContainerStarted","Data":"c2003f19cf7d6a4f05f4fb76fb6a106b857e911e3f85ecc5d14a785250bcf519"} Oct 29 23:09:29.855670 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:29.855592 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" podStartSLOduration=4.855573102 podStartE2EDuration="4.855573102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:29.853664803 +0000 UTC m=+1623.794212714" watchObservedRunningTime="2026-10-29 23:09:29.855573102 +0000 UTC m=+1623.796121014" Oct 29 23:09:29.916706 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:29.916570 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" podStartSLOduration=4.916552696 podStartE2EDuration="4.916552696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:29.915737735 +0000 UTC m=+1623.856285647" watchObservedRunningTime="2026-10-29 23:09:29.916552696 +0000 UTC m=+1623.857100607" Oct 29 23:09:29.984602 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:29.984566 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 29 23:09:29.984889 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:29.984860 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" podUID="2f71bdab-b1a2-4513-a7f1-2455e4222efa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 29 23:09:30.294611 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:30.289572 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:30.294611 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:30.292835 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 29 23:09:30.294611 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:30.292910 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" podUID="2cdd98c2-c079-4880-9835-36d2c4a86de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 29 23:09:30.516227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:30.515057 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:30.537239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:30.526553 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Oct 29 23:09:30.537239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:30.526619 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" podUID="be674d35-b876-4f7d-9ceb-8cd285a3f2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Oct 29 23:09:30.812609 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:30.812578 4821 generic.go:356] "Generic (PLEG): container finished" podID="d7d413f3-e64a-4367-948a-5d96ff9775de" containerID="afd932e4286570989947c901d58b5dd981d6239d6bb6f4aba380f172ea78732c" exitCode=0 Oct 29 23:09:30.813005 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:30.812973 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f36035d-zft54" event={"ID":"d7d413f3-e64a-4367-948a-5d96ff9775de","Type":"ContainerDied","Data":"afd932e4286570989947c901d58b5dd981d6239d6bb6f4aba380f172ea78732c"} Oct 29 23:09:30.987720 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:30.987622 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 29 23:09:30.987982 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:30.987953 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" podUID="2f71bdab-b1a2-4513-a7f1-2455e4222efa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 29 23:09:31.298407 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:31.293653 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 29 23:09:31.298407 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:31.293732 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" podUID="2cdd98c2-c079-4880-9835-36d2c4a86de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 29 23:09:31.492526 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:31.492491 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Oct 29 23:09:31.492827 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:31.492784 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" podUID="be674d35-b876-4f7d-9ceb-8cd285a3f2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Oct 29 23:09:32.000269 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:31.990591 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 29 23:09:32.000269 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:31.990652 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" podUID="2f71bdab-b1a2-4513-a7f1-2455e4222efa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 29 23:09:32.305006 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:32.303944 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 29 23:09:32.305006 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:32.304001 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" podUID="2cdd98c2-c079-4880-9835-36d2c4a86de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 29 23:09:32.490353 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:32.488751 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Oct 29 23:09:32.490353 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:32.488816 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" podUID="be674d35-b876-4f7d-9ceb-8cd285a3f2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Oct 29 23:09:32.583389 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:32.583361 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f36035d-zft54" Oct 29 23:09:32.606916 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:32.606890 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7d413f3-e64a-4367-948a-5d96ff9775de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d413f3-e64a-4367-948a-5d96ff9775de-maas-api-xks-ca\") pod \"d7d413f3-e64a-4367-948a-5d96ff9775de\" (UID: \"d7d413f3-e64a-4367-948a-5d96ff9775de\") " Oct 29 23:09:32.607079 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:32.606967 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d413f3-e64a-4367-948a-5d96ff9775de-kube-api-access-p9h8k\" (UniqueName: \"kubernetes.io/projected/d7d413f3-e64a-4367-948a-5d96ff9775de-kube-api-access-p9h8k\") pod \"d7d413f3-e64a-4367-948a-5d96ff9775de\" (UID: \"d7d413f3-e64a-4367-948a-5d96ff9775de\") " Oct 29 23:09:32.607079 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:32.607004 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7d413f3-e64a-4367-948a-5d96ff9775de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d413f3-e64a-4367-948a-5d96ff9775de-maas-api-openshift-ca\") pod \"d7d413f3-e64a-4367-948a-5d96ff9775de\" (UID: \"d7d413f3-e64a-4367-948a-5d96ff9775de\") " Oct 29 23:09:32.607414 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:32.607390 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d413f3-e64a-4367-948a-5d96ff9775de-maas-api-openshift-ca" pod "d7d413f3-e64a-4367-948a-5d96ff9775de" (UID: "d7d413f3-e64a-4367-948a-5d96ff9775de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:09:32.607619 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:32.607571 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d413f3-e64a-4367-948a-5d96ff9775de-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:09:32.609727 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:32.609693 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d413f3-e64a-4367-948a-5d96ff9775de-maas-api-xks-ca" pod "d7d413f3-e64a-4367-948a-5d96ff9775de" (UID: "d7d413f3-e64a-4367-948a-5d96ff9775de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:09:32.609805 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:32.609708 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d413f3-e64a-4367-948a-5d96ff9775de-kube-api-access-p9h8k" pod "d7d413f3-e64a-4367-948a-5d96ff9775de" (UID: "d7d413f3-e64a-4367-948a-5d96ff9775de"). InnerVolumeSpecName "kube-api-access-p9h8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:09:32.752288 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:32.724716 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9h8k\" (UniqueName: \"kubernetes.io/projected/d7d413f3-e64a-4367-948a-5d96ff9775de-kube-api-access-p9h8k\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:09:32.752594 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:32.752568 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d413f3-e64a-4367-948a-5d96ff9775de-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:09:32.895440 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:32.895410 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f36035d-zft54" Oct 29 23:09:32.895440 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:32.895428 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f36035d-zft54" event={"ID":"d7d413f3-e64a-4367-948a-5d96ff9775de","Type":"ContainerDied","Data":"0de46000b6e40696ec110dee879176c9e192ccf14b1fcf07698f28e5e6ef0e52"} Oct 29 23:09:32.895688 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:32.895491 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0de46000b6e40696ec110dee879176c9e192ccf14b1fcf07698f28e5e6ef0e52" Oct 29 23:09:32.982285 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:32.982250 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 29 23:09:32.982444 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:32.982316 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" podUID="2f71bdab-b1a2-4513-a7f1-2455e4222efa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 29 23:09:33.287892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:33.287855 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 29 23:09:33.288349 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:33.287905 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" podUID="2cdd98c2-c079-4880-9835-36d2c4a86de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 29 23:09:33.487527 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:33.487445 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Oct 29 23:09:33.487706 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:33.487548 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" podUID="be674d35-b876-4f7d-9ceb-8cd285a3f2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Oct 29 23:09:34.292228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:34.291839 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 29 23:09:34.292228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:34.291912 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" podUID="2cdd98c2-c079-4880-9835-36d2c4a86de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 29 23:09:34.488269 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:34.488190 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Oct 29 23:09:34.488496 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:34.488278 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" podUID="be674d35-b876-4f7d-9ceb-8cd285a3f2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Oct 29 23:09:34.497553 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:34.497227 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:34.498292 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:34.498264 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:34.500137 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:34.500111 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:09:35.287977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:35.287941 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 29 23:09:35.288228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:35.288012 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" podUID="2cdd98c2-c079-4880-9835-36d2c4a86de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 29 23:09:35.500225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:35.499993 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Oct 29 23:09:35.500225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:35.500050 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" podUID="be674d35-b876-4f7d-9ceb-8cd285a3f2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Oct 29 23:09:36.288515 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:36.288473 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 29 23:09:36.288830 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:36.288788 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" podUID="2cdd98c2-c079-4880-9835-36d2c4a86de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 29 23:09:36.486967 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:36.486847 4821 patch_prober.go:29] interesting pod/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Oct 29 23:09:36.486967 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:36.486920 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" podUID="be674d35-b876-4f7d-9ceb-8cd285a3f2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Oct 29 23:09:37.405806 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:37.405769 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:37.406715 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:37.406689 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:37.407424 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:37.407401 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:09:37.491089 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:37.491057 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:37.491511 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:37.491485 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:37.492107 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:37.492090 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:09:40.494384 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:40.494350 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a004662-7nj9d"] Oct 29 23:09:40.495207 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:40.495167 4821 cpu_manager.go:414] "containerMap: removing container" podUID="d7d413f3-e64a-4367-948a-5d96ff9775de" containerName="revoke-keys" Oct 29 23:09:40.495387 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:40.495225 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d413f3-e64a-4367-948a-5d96ff9775de" containerName="revoke-keys" Oct 29 23:09:40.495452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:40.495425 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d413f3-e64a-4367-948a-5d96ff9775de" containerName="revoke-keys" Oct 29 23:09:40.498671 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:40.498650 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a004662-7nj9d" Oct 29 23:09:40.521847 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:40.521816 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:09:40.525650 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:40.525622 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a004662-7nj9d"] Oct 29 23:09:40.529433 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:40.529412 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:09:40.628127 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:40.628092 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9k5l\" (UniqueName: \"kubernetes.io/projected/9d7b420c-09a6-4125-82df-699549ab29ee-kube-api-access-q9k5l\") pod \"maas-api-revoke-sub-8a004662-7nj9d\" (UID: \"9d7b420c-09a6-4125-82df-699549ab29ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a004662-7nj9d" Oct 29 23:09:40.628345 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:40.628236 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d7b420c-09a6-4125-82df-699549ab29ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a004662-7nj9d\" (UID: \"9d7b420c-09a6-4125-82df-699549ab29ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a004662-7nj9d" Oct 29 23:09:40.628345 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:40.628268 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d7b420c-09a6-4125-82df-699549ab29ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a004662-7nj9d\" (UID: \"9d7b420c-09a6-4125-82df-699549ab29ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a004662-7nj9d" Oct 29 23:09:40.729305 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:40.729273 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9k5l\" (UniqueName: \"kubernetes.io/projected/9d7b420c-09a6-4125-82df-699549ab29ee-kube-api-access-q9k5l\") pod \"maas-api-revoke-sub-8a004662-7nj9d\" (UID: \"9d7b420c-09a6-4125-82df-699549ab29ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a004662-7nj9d" Oct 29 23:09:40.729487 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:40.729323 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d7b420c-09a6-4125-82df-699549ab29ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a004662-7nj9d\" (UID: \"9d7b420c-09a6-4125-82df-699549ab29ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a004662-7nj9d" Oct 29 23:09:40.729487 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:40.729345 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d7b420c-09a6-4125-82df-699549ab29ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a004662-7nj9d\" (UID: \"9d7b420c-09a6-4125-82df-699549ab29ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a004662-7nj9d" Oct 29 23:09:40.729910 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:40.729890 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d7b420c-09a6-4125-82df-699549ab29ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a004662-7nj9d\" (UID: \"9d7b420c-09a6-4125-82df-699549ab29ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a004662-7nj9d" Oct 29 23:09:40.731948 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:40.731924 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d7b420c-09a6-4125-82df-699549ab29ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a004662-7nj9d\" (UID: \"9d7b420c-09a6-4125-82df-699549ab29ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a004662-7nj9d" Oct 29 23:09:40.753793 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:40.753733 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9k5l\" (UniqueName: \"kubernetes.io/projected/9d7b420c-09a6-4125-82df-699549ab29ee-kube-api-access-q9k5l\") pod \"maas-api-revoke-sub-8a004662-7nj9d\" (UID: \"9d7b420c-09a6-4125-82df-699549ab29ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a004662-7nj9d" Oct 29 23:09:40.814860 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:40.814834 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a004662-7nj9d" Oct 29 23:09:41.130330 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:41.130300 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a004662-7nj9d"] Oct 29 23:09:41.131411 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:09:41.131382 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d7b420c_09a6_4125_82df_699549ab29ee.slice/crio-70e94cc45f24b52240d1a40df7d9f0f293e7ca760b049862b991a5d6a30f1557 WatchSource:0}: Error finding container 70e94cc45f24b52240d1a40df7d9f0f293e7ca760b049862b991a5d6a30f1557: Status 404 returned error can't find the container with id 70e94cc45f24b52240d1a40df7d9f0f293e7ca760b049862b991a5d6a30f1557 Oct 29 23:09:41.987855 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:41.987819 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a004662-7nj9d" event={"ID":"9d7b420c-09a6-4125-82df-699549ab29ee","Type":"ContainerStarted","Data":"ca4df17ab31c988fd8078c36c196438dbbe5fadbb54a5f3ac166e5d3deb7bbaf"} Oct 29 23:09:41.988347 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:41.987874 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a004662-7nj9d" event={"ID":"9d7b420c-09a6-4125-82df-699549ab29ee","Type":"ContainerStarted","Data":"70e94cc45f24b52240d1a40df7d9f0f293e7ca760b049862b991a5d6a30f1557"} Oct 29 23:09:42.034599 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:42.034548 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a004662-7nj9d" podStartSLOduration=2.034529657 podStartE2EDuration="2.034529657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:42.031899942 +0000 UTC m=+1635.972447853" watchObservedRunningTime="2026-10-29 23:09:42.034529657 +0000 UTC m=+1635.975077566" Oct 29 23:09:42.996662 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:42.996628 4821 generic.go:356] "Generic (PLEG): container finished" podID="9d7b420c-09a6-4125-82df-699549ab29ee" containerID="ca4df17ab31c988fd8078c36c196438dbbe5fadbb54a5f3ac166e5d3deb7bbaf" exitCode=0 Oct 29 23:09:42.997091 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:42.996713 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a004662-7nj9d" event={"ID":"9d7b420c-09a6-4125-82df-699549ab29ee","Type":"ContainerDied","Data":"ca4df17ab31c988fd8078c36c196438dbbe5fadbb54a5f3ac166e5d3deb7bbaf"} Oct 29 23:09:44.182546 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:44.182520 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a004662-7nj9d" Oct 29 23:09:44.296500 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:44.296417 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d7b420c-09a6-4125-82df-699549ab29ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d7b420c-09a6-4125-82df-699549ab29ee-maas-api-xks-ca\") pod \"9d7b420c-09a6-4125-82df-699549ab29ee\" (UID: \"9d7b420c-09a6-4125-82df-699549ab29ee\") " Oct 29 23:09:44.296633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:44.296542 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d7b420c-09a6-4125-82df-699549ab29ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d7b420c-09a6-4125-82df-699549ab29ee-maas-api-openshift-ca\") pod \"9d7b420c-09a6-4125-82df-699549ab29ee\" (UID: \"9d7b420c-09a6-4125-82df-699549ab29ee\") " Oct 29 23:09:44.296633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:44.296588 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d7b420c-09a6-4125-82df-699549ab29ee-kube-api-access-q9k5l\" (UniqueName: \"kubernetes.io/projected/9d7b420c-09a6-4125-82df-699549ab29ee-kube-api-access-q9k5l\") pod \"9d7b420c-09a6-4125-82df-699549ab29ee\" (UID: \"9d7b420c-09a6-4125-82df-699549ab29ee\") " Oct 29 23:09:44.296897 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:44.296866 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d7b420c-09a6-4125-82df-699549ab29ee-maas-api-openshift-ca" pod "9d7b420c-09a6-4125-82df-699549ab29ee" (UID: "9d7b420c-09a6-4125-82df-699549ab29ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:09:44.297116 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:44.297101 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d7b420c-09a6-4125-82df-699549ab29ee-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:09:44.298889 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:44.298866 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d7b420c-09a6-4125-82df-699549ab29ee-kube-api-access-q9k5l" pod "9d7b420c-09a6-4125-82df-699549ab29ee" (UID: "9d7b420c-09a6-4125-82df-699549ab29ee"). InnerVolumeSpecName "kube-api-access-q9k5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:09:44.298974 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:44.298866 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d7b420c-09a6-4125-82df-699549ab29ee-maas-api-xks-ca" pod "9d7b420c-09a6-4125-82df-699549ab29ee" (UID: "9d7b420c-09a6-4125-82df-699549ab29ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:09:44.399045 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:44.399014 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9k5l\" (UniqueName: \"kubernetes.io/projected/9d7b420c-09a6-4125-82df-699549ab29ee-kube-api-access-q9k5l\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:09:44.399045 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:44.399042 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d7b420c-09a6-4125-82df-699549ab29ee-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:09:45.014750 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:45.014723 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a004662-7nj9d" Oct 29 23:09:45.014905 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:45.014756 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a004662-7nj9d" event={"ID":"9d7b420c-09a6-4125-82df-699549ab29ee","Type":"ContainerDied","Data":"70e94cc45f24b52240d1a40df7d9f0f293e7ca760b049862b991a5d6a30f1557"} Oct 29 23:09:45.014905 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:45.014804 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70e94cc45f24b52240d1a40df7d9f0f293e7ca760b049862b991a5d6a30f1557" Oct 29 23:09:51.004043 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:51.004009 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s"] Oct 29 23:09:51.004571 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:51.004555 4821 cpu_manager.go:414] "containerMap: removing container" podUID="9d7b420c-09a6-4125-82df-699549ab29ee" containerName="revoke-keys" Oct 29 23:09:51.004618 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:51.004572 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d7b420c-09a6-4125-82df-699549ab29ee" containerName="revoke-keys" Oct 29 23:09:51.004706 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:51.004697 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d7b420c-09a6-4125-82df-699549ab29ee" containerName="revoke-keys" Oct 29 23:09:51.008187 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:51.008170 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" Oct 29 23:09:51.027425 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:51.027108 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-b0c3c3-serving-cert\"" Oct 29 23:09:51.028084 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:51.027911 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s"] Oct 29 23:09:51.093108 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:51.093073 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47a79417-347e-49b8-b4b5-45896eccdde2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s\" (UID: \"47a79417-347e-49b8-b4b5-45896eccdde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" Oct 29 23:09:51.093341 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:51.093230 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zndmd\" (UniqueName: \"kubernetes.io/projected/47a79417-347e-49b8-b4b5-45896eccdde2-kube-api-access-zndmd\") pod \"maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s\" (UID: \"47a79417-347e-49b8-b4b5-45896eccdde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" Oct 29 23:09:51.093341 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:51.093320 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47a79417-347e-49b8-b4b5-45896eccdde2-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s\" (UID: \"47a79417-347e-49b8-b4b5-45896eccdde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" Oct 29 23:09:51.093456 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:51.093357 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47a79417-347e-49b8-b4b5-45896eccdde2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s\" (UID: \"47a79417-347e-49b8-b4b5-45896eccdde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" Oct 29 23:09:51.203255 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:51.203161 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47a79417-347e-49b8-b4b5-45896eccdde2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s\" (UID: \"47a79417-347e-49b8-b4b5-45896eccdde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" Oct 29 23:09:51.203478 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:51.203364 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zndmd\" (UniqueName: \"kubernetes.io/projected/47a79417-347e-49b8-b4b5-45896eccdde2-kube-api-access-zndmd\") pod \"maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s\" (UID: \"47a79417-347e-49b8-b4b5-45896eccdde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" Oct 29 23:09:51.203683 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:51.203485 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47a79417-347e-49b8-b4b5-45896eccdde2-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s\" (UID: \"47a79417-347e-49b8-b4b5-45896eccdde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" Oct 29 23:09:51.203683 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:51.203520 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47a79417-347e-49b8-b4b5-45896eccdde2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s\" (UID: \"47a79417-347e-49b8-b4b5-45896eccdde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" Oct 29 23:09:51.207060 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:51.207026 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47a79417-347e-49b8-b4b5-45896eccdde2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s\" (UID: \"47a79417-347e-49b8-b4b5-45896eccdde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" Oct 29 23:09:51.213246 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:51.213156 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47a79417-347e-49b8-b4b5-45896eccdde2-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s\" (UID: \"47a79417-347e-49b8-b4b5-45896eccdde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" Oct 29 23:09:51.213405 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:51.213302 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47a79417-347e-49b8-b4b5-45896eccdde2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s\" (UID: \"47a79417-347e-49b8-b4b5-45896eccdde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" Oct 29 23:09:51.231102 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:51.231069 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zndmd\" (UniqueName: \"kubernetes.io/projected/47a79417-347e-49b8-b4b5-45896eccdde2-kube-api-access-zndmd\") pod \"maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s\" (UID: \"47a79417-347e-49b8-b4b5-45896eccdde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" Oct 29 23:09:51.322723 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:51.322624 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" Oct 29 23:09:51.645753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:51.645724 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s"] Oct 29 23:09:51.646233 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:09:51.646153 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a79417_347e_49b8_b4b5_45896eccdde2.slice/crio-8248e8cae6d828dbd6fea38cda8d53916e1bfc5cf220acd72298d3a6e73a2e88 WatchSource:0}: Error finding container 8248e8cae6d828dbd6fea38cda8d53916e1bfc5cf220acd72298d3a6e73a2e88: Status 404 returned error can't find the container with id 8248e8cae6d828dbd6fea38cda8d53916e1bfc5cf220acd72298d3a6e73a2e88 Oct 29 23:09:51.647335 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:51.647317 4821 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:09:52.077381 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.077347 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" event={"ID":"47a79417-347e-49b8-b4b5-45896eccdde2","Type":"ContainerStarted","Data":"8248e8cae6d828dbd6fea38cda8d53916e1bfc5cf220acd72298d3a6e73a2e88"} Oct 29 23:09:52.278129 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.278043 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b"] Oct 29 23:09:52.283751 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.283722 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.302090 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.302055 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr"] Oct 29 23:09:52.307630 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.307599 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr"] Oct 29 23:09:52.307796 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.307712 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.314667 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.314632 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b"] Oct 29 23:09:52.322419 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.322389 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.322621 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.322601 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.322776 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.322759 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.322920 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.322904 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137c7cf5-e922-49d1-acbc-fc0b13899da5-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.323239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.323182 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.323481 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.323451 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fblfg\" (UniqueName: \"kubernetes.io/projected/137c7cf5-e922-49d1-acbc-fc0b13899da5-kube-api-access-fblfg\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.323631 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.323616 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.323879 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.323858 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.324019 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.324004 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.425663 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.425625 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.425892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.425675 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137c7cf5-e922-49d1-acbc-fc0b13899da5-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.425892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.425719 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0da6fd9-efd5-4587-b764-524010b6a352-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.425892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.425745 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0da6fd9-efd5-4587-b764-524010b6a352-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.425892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.425787 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.425892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.425866 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.426181 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.425903 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.426181 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.425924 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfv6g\" (UniqueName: \"kubernetes.io/projected/c0da6fd9-efd5-4587-b764-524010b6a352-kube-api-access-lfv6g\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.426181 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.425953 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.426181 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.426004 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fblfg\" (UniqueName: \"kubernetes.io/projected/137c7cf5-e922-49d1-acbc-fc0b13899da5-kube-api-access-fblfg\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.426181 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.426021 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.426181 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.426051 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.426181 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.426085 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.426181 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.426106 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0da6fd9-efd5-4587-b764-524010b6a352-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.426181 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.426156 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.426753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.426194 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.426753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.426260 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.426753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.426288 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.426753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.426514 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137c7cf5-e922-49d1-acbc-fc0b13899da5-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.426753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.426690 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.427314 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.427184 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.427314 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.427277 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.427314 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.427296 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.428859 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.428836 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.429515 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.429492 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.456098 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.456038 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fblfg\" (UniqueName: \"kubernetes.io/projected/137c7cf5-e922-49d1-acbc-fc0b13899da5-kube-api-access-fblfg\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.456830 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.456803 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.533288 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.533140 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0da6fd9-efd5-4587-b764-524010b6a352-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.533288 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.533271 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0da6fd9-efd5-4587-b764-524010b6a352-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.533546 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.533329 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.533546 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.533392 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.533546 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.533421 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfv6g\" (UniqueName: \"kubernetes.io/projected/c0da6fd9-efd5-4587-b764-524010b6a352-kube-api-access-lfv6g\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.533546 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.533462 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.533546 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.533542 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.533828 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.533612 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.533828 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.533645 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0da6fd9-efd5-4587-b764-524010b6a352-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.542734 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.541069 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0da6fd9-efd5-4587-b764-524010b6a352-istiod-ca-cert\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.542734 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.541405 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-istio-data\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.542734 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.541645 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-workload-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.542734 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.542242 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-workload-certs\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.542734 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.542488 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-credential-socket\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.566861 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.566825 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0da6fd9-efd5-4587-b764-524010b6a352-istio-podinfo\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.567554 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.567527 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0da6fd9-efd5-4587-b764-524010b6a352-istio-token\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.580307 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.578617 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-istio-envoy\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.610819 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.610777 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfv6g\" (UniqueName: \"kubernetes.io/projected/c0da6fd9-efd5-4587-b764-524010b6a352-kube-api-access-lfv6g\") pod \"e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.611081 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.611061 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:52.679870 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.673341 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:52.789478 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.789445 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2"] Oct 29 23:09:52.795521 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.795495 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:52.814042 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.813381 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-65e2d3-openshift-default-dockercfg-w5gd5\"" Oct 29 23:09:52.820972 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.820925 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2"] Oct 29 23:09:52.881226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.880940 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7msqw\" (UniqueName: \"kubernetes.io/projected/f3f2d74f-e211-41ce-8ed6-432892f64365-kube-api-access-7msqw\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:52.881226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.881073 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-token\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:52.881226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.881114 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-credential-socket\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:52.881226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.881184 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-workload-certs\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:52.881600 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.881263 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-envoy\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:52.881600 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.881290 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-podinfo\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:52.881600 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.881391 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-data\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:52.881600 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.881484 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f2d74f-e211-41ce-8ed6-432892f64365-istiod-ca-cert\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:52.881820 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.881612 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-workload-socket\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:52.987812 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.987775 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-credential-socket\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:52.987996 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.987865 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-workload-certs\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:52.987996 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.987926 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-envoy\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:52.987996 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.987952 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-podinfo\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:52.988164 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.988004 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-data\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:52.988164 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.988097 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f2d74f-e211-41ce-8ed6-432892f64365-istiod-ca-cert\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:52.988303 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.988262 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-workload-socket\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:52.992857 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.992821 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7msqw\" (UniqueName: \"kubernetes.io/projected/f3f2d74f-e211-41ce-8ed6-432892f64365-kube-api-access-7msqw\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:52.993032 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.992962 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-token\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:52.997233 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.994834 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f2d74f-e211-41ce-8ed6-432892f64365-istiod-ca-cert\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:52.997233 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.995065 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-data\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:52.997233 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.995323 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-credential-socket\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:52.997518 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.997329 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-workload-socket\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:52.997518 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.997406 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-workload-certs\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:53.013349 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:52.998109 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-envoy\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:53.013349 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.008944 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-podinfo\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:53.022652 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.022584 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7msqw\" (UniqueName: \"kubernetes.io/projected/f3f2d74f-e211-41ce-8ed6-432892f64365-kube-api-access-7msqw\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:53.131308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.080828 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-token\") pod \"e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:53.145814 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.135258 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:53.175713 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.173556 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" event={"ID":"47a79417-347e-49b8-b4b5-45896eccdde2","Type":"ContainerStarted","Data":"e6be7b758c0d252e76914a2340bb6714310860a9ea47e47a0ac62fad7e13a597"} Oct 29 23:09:53.175713 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.174548 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" Oct 29 23:09:53.201578 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.190485 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b"] Oct 29 23:09:53.206119 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.205227 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:09:53.206119 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.205329 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:09:53.206119 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.205375 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:09:53.228244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.225297 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" podStartSLOduration=2.8743179530000003 podStartE2EDuration="3.225279059s" totalImagesPullingTime="350.961106ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:53.223719754 +0000 UTC m=+1647.164267673" watchObservedRunningTime="2026-10-29 23:09:53.225279059 +0000 UTC m=+1647.165826969" Oct 29 23:09:53.488330 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:09:53.488255 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0da6fd9_efd5_4587_b764_524010b6a352.slice/crio-15c3d6a5e3c7d82f14fac836c1b354494a22a2ec127854ced69495706f25eb31 WatchSource:0}: Error finding container 15c3d6a5e3c7d82f14fac836c1b354494a22a2ec127854ced69495706f25eb31: Status 404 returned error can't find the container with id 15c3d6a5e3c7d82f14fac836c1b354494a22a2ec127854ced69495706f25eb31 Oct 29 23:09:53.489085 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.488598 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr"] Oct 29 23:09:53.511549 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.511505 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:09:53.511672 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.511595 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:09:53.511672 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.511638 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:09:53.526531 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.525246 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj"] Oct 29 23:09:53.534383 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.534347 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.558506 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.558478 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-67363b-openshift-default-dockercfg-kr7r9\"" Oct 29 23:09:53.570702 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.570665 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj"] Oct 29 23:09:53.611249 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.607104 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:09:53.611249 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.607193 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:09:53.616247 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.612922 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:09:53.616247 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.610115 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-envoy\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.616247 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.613512 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-workload-certs\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.616247 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.613798 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-credential-socket\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.616247 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.613914 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-workload-socket\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.616247 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.613961 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rbz9\" (UniqueName: \"kubernetes.io/projected/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-kube-api-access-5rbz9\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.616247 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.614280 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-token\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.616247 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.614330 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istiod-ca-cert\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.616247 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.614486 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-data\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.616247 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.614537 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-podinfo\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.688883 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.688178 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2"] Oct 29 23:09:53.717175 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.717135 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-envoy\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.717322 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.717274 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-workload-certs\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.717401 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.717344 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-credential-socket\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.717460 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.717411 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-workload-socket\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.717523 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.717455 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rbz9\" (UniqueName: \"kubernetes.io/projected/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-kube-api-access-5rbz9\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.717523 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.717516 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-token\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.717649 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.717543 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istiod-ca-cert\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.717649 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.717596 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-data\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.717649 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.717631 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-podinfo\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.718591 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.718556 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-workload-socket\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.720884 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.719903 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istiod-ca-cert\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.721034 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.720128 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-data\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.721136 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.720346 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-workload-certs\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.721238 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.720831 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-credential-socket\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.752570 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.752540 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-token\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.752955 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.752850 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rbz9\" (UniqueName: \"kubernetes.io/projected/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-kube-api-access-5rbz9\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.755608 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.755581 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-podinfo\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.756917 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.756893 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-envoy\") pod \"e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.903303 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:53.903267 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:53.957654 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:09:53.957616 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:09:54.186750 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:54.186718 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" event={"ID":"c0da6fd9-efd5-4587-b764-524010b6a352","Type":"ContainerStarted","Data":"509e9a78c01a9d34f3ca79be700e171ac50110f12c53a9e10b3d2fa1181d25a2"} Oct 29 23:09:54.187486 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:54.186769 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" event={"ID":"c0da6fd9-efd5-4587-b764-524010b6a352","Type":"ContainerStarted","Data":"15c3d6a5e3c7d82f14fac836c1b354494a22a2ec127854ced69495706f25eb31"} Oct 29 23:09:54.188935 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:54.188897 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" event={"ID":"137c7cf5-e922-49d1-acbc-fc0b13899da5","Type":"ContainerStarted","Data":"5fa4fac44d9dc85377625cb0f50c6c1f1039ed1b104e3ed81de01b4c0e7b3eca"} Oct 29 23:09:54.189050 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:54.188949 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" event={"ID":"137c7cf5-e922-49d1-acbc-fc0b13899da5","Type":"ContainerStarted","Data":"5ef2c6b4f114e5c2c65856558c3f5ec55c9ab0963a51134df511389410eea6af"} Oct 29 23:09:54.191004 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:54.190980 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" event={"ID":"f3f2d74f-e211-41ce-8ed6-432892f64365","Type":"ContainerStarted","Data":"646da0945b2d2ef89e14a88a15f2757d0928bb9e4ad2381bf453824fa47e4184"} Oct 29 23:09:54.191112 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:54.191025 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" event={"ID":"f3f2d74f-e211-41ce-8ed6-432892f64365","Type":"ContainerStarted","Data":"a55aca95be1da619929763b72991c43cc2379d4b8d4e85f3edc680da52d58d4e"} Oct 29 23:09:54.240255 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:54.240176 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" podStartSLOduration=2.240159197 podStartE2EDuration="2.240159197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:54.235428854 +0000 UTC m=+1648.175976765" watchObservedRunningTime="2026-10-29 23:09:54.240159197 +0000 UTC m=+1648.180707106" Oct 29 23:09:54.287813 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:54.287705 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" podStartSLOduration=2.287689721 podStartE2EDuration="2.287689721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:54.285485693 +0000 UTC m=+1648.226033602" watchObservedRunningTime="2026-10-29 23:09:54.287689721 +0000 UTC m=+1648.228237686" Oct 29 23:09:54.319407 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:54.319371 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj"] Oct 29 23:09:54.340544 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:54.340498 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:09:54.340794 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:54.340769 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:09:54.340951 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:54.340931 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:09:54.370101 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:54.370019 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" podStartSLOduration=2.369999048 podStartE2EDuration="2.369999048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:54.364541796 +0000 UTC m=+1648.305089707" watchObservedRunningTime="2026-10-29 23:09:54.369999048 +0000 UTC m=+1648.310546959" Oct 29 23:09:54.612359 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:54.612282 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:09:54.613722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:54.613694 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Oct 29 23:09:54.613879 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:54.613765 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" podUID="137c7cf5-e922-49d1-acbc-fc0b13899da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Oct 29 23:09:54.681443 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:54.681417 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:09:54.682742 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:54.682710 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Oct 29 23:09:54.682872 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:54.682770 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" podUID="c0da6fd9-efd5-4587-b764-524010b6a352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Oct 29 23:09:55.151226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:55.147874 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:55.173226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:55.164967 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:55.202445 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:55.202355 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" event={"ID":"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a","Type":"ContainerStarted","Data":"d1217ac883fb396639158d9a394a35279bb12c6ba71cb0290eafd90e822f2101"} Oct 29 23:09:55.202445 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:55.202414 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" event={"ID":"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a","Type":"ContainerStarted","Data":"4b1ca03a34470000e79fb4c5e4942c83785d62c30599e716fe925b2eb1da75eb"} Oct 29 23:09:55.203091 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:55.203028 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:55.209271 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:55.205920 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:09:55.362226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:55.354674 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" podStartSLOduration=2.35465182 podStartE2EDuration="2.35465182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:55.289264115 +0000 UTC m=+1649.229812024" watchObservedRunningTime="2026-10-29 23:09:55.35465182 +0000 UTC m=+1649.295199730" Oct 29 23:09:55.692381 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:55.692343 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Oct 29 23:09:55.692678 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:55.692647 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" podUID="137c7cf5-e922-49d1-acbc-fc0b13899da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Oct 29 23:09:55.701927 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:55.701900 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Oct 29 23:09:55.702156 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:55.702135 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" podUID="c0da6fd9-efd5-4587-b764-524010b6a352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Oct 29 23:09:55.904089 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:55.904062 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:55.909925 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:55.909900 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:56.213333 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:56.213302 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:56.222226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:56.220496 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:09:56.613269 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:56.613236 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Oct 29 23:09:56.613438 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:56.613293 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" podUID="137c7cf5-e922-49d1-acbc-fc0b13899da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Oct 29 23:09:56.681866 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:56.681829 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Oct 29 23:09:56.682126 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:56.682099 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" podUID="c0da6fd9-efd5-4587-b764-524010b6a352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Oct 29 23:09:57.611784 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:57.611741 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Oct 29 23:09:57.612385 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:57.611804 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" podUID="137c7cf5-e922-49d1-acbc-fc0b13899da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Oct 29 23:09:57.681229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:57.680885 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Oct 29 23:09:57.681229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:57.680962 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" podUID="c0da6fd9-efd5-4587-b764-524010b6a352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Oct 29 23:09:58.612334 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:58.612300 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Oct 29 23:09:58.612919 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:58.612348 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" podUID="137c7cf5-e922-49d1-acbc-fc0b13899da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Oct 29 23:09:58.680697 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:58.680659 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Oct 29 23:09:58.680884 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:58.680715 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" podUID="c0da6fd9-efd5-4587-b764-524010b6a352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Oct 29 23:09:59.492883 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.492835 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw"] Oct 29 23:09:59.499922 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.499888 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.526817 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.526789 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-3e5b21-openshift-default-dockercfg-4zmbk\"" Oct 29 23:09:59.595345 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.595307 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw"] Oct 29 23:09:59.605090 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.604125 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-envoy\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.608541 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.608500 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-workload-certs\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.609501 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.609472 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-token\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.640945 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.609729 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-credential-socket\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.656919 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.656874 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-data\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.657231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.657184 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78z9f\" (UniqueName: \"kubernetes.io/projected/b74847df-b26e-4c6b-8f86-b86b5282c004-kube-api-access-78z9f\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.657481 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.657464 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-podinfo\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.671331 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.671295 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-workload-socket\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.671681 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.671660 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74847df-b26e-4c6b-8f86-b86b5282c004-istiod-ca-cert\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.699250 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.675243 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Oct 29 23:09:59.699250 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.675305 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" podUID="137c7cf5-e922-49d1-acbc-fc0b13899da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Oct 29 23:09:59.699250 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.696752 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Oct 29 23:09:59.699250 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.696806 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" podUID="c0da6fd9-efd5-4587-b764-524010b6a352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Oct 29 23:09:59.778280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.777677 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-envoy\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.778280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.777825 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-workload-certs\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.778280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.777854 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-token\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.778280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.777909 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-credential-socket\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.778280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.777979 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-data\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.778280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.778015 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78z9f\" (UniqueName: \"kubernetes.io/projected/b74847df-b26e-4c6b-8f86-b86b5282c004-kube-api-access-78z9f\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.778280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.778125 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-podinfo\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.778280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.778174 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-workload-socket\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.778871 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.778307 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74847df-b26e-4c6b-8f86-b86b5282c004-istiod-ca-cert\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.815848 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.779167 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74847df-b26e-4c6b-8f86-b86b5282c004-istiod-ca-cert\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.815848 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.803024 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-workload-certs\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.815848 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.804993 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-data\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.819390 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.819356 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-podinfo\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.819726 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.819706 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-credential-socket\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.820164 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.819947 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-workload-socket\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.820316 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.820116 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-envoy\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.844059 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.843993 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-token\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.888245 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.865771 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78z9f\" (UniqueName: \"kubernetes.io/projected/b74847df-b26e-4c6b-8f86-b86b5282c004-kube-api-access-78z9f\") pod \"e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:09:59.918909 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:09:59.912499 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:10:00.296227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:00.295557 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" Oct 29 23:10:00.630373 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:00.630330 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:00.656307 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:00.636570 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Oct 29 23:10:00.656307 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:00.636622 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" podUID="137c7cf5-e922-49d1-acbc-fc0b13899da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Oct 29 23:10:00.656307 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:00.640429 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:00.700469 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:00.700433 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Oct 29 23:10:00.700663 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:00.700507 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" podUID="c0da6fd9-efd5-4587-b764-524010b6a352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Oct 29 23:10:00.737439 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:00.737358 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw"] Oct 29 23:10:00.797285 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:00.788680 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:10:00.797285 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:00.788784 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:10:00.797285 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:00.788829 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:10:01.594274 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:01.585387 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" event={"ID":"b74847df-b26e-4c6b-8f86-b86b5282c004","Type":"ContainerStarted","Data":"cc6eafa394643addd478306d1c2e1502129939e76105f64d7a2c3e05e9eb7c21"} Oct 29 23:10:01.637232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:01.633862 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Oct 29 23:10:01.637232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:01.633940 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" podUID="137c7cf5-e922-49d1-acbc-fc0b13899da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Oct 29 23:10:01.690229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:01.683336 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Oct 29 23:10:01.690229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:01.683412 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" podUID="c0da6fd9-efd5-4587-b764-524010b6a352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Oct 29 23:10:02.617537 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:02.616589 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Oct 29 23:10:02.617537 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:02.616655 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" podUID="137c7cf5-e922-49d1-acbc-fc0b13899da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Oct 29 23:10:02.651506 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:02.650406 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" event={"ID":"b74847df-b26e-4c6b-8f86-b86b5282c004","Type":"ContainerStarted","Data":"3f0561ec234de9dda8290ebc9313b4559c76f24e3d91a44c7b6ba8ae61f87ae0"} Oct 29 23:10:02.683765 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:02.683732 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Oct 29 23:10:02.684085 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:02.684037 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" podUID="c0da6fd9-efd5-4587-b764-524010b6a352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Oct 29 23:10:02.857229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:02.834902 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" podStartSLOduration=3.834877906 podStartE2EDuration="3.834877906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:02.720515832 +0000 UTC m=+1656.661063743" watchObservedRunningTime="2026-10-29 23:10:02.834877906 +0000 UTC m=+1656.775425818" Oct 29 23:10:02.917364 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:02.917274 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:10:02.991628 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:02.991595 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:10:03.599511 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:03.598497 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:03.647877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:03.647828 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:03.684231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:03.672170 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:03.688045 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:03.686322 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Oct 29 23:10:03.688313 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:03.688274 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" podUID="c0da6fd9-efd5-4587-b764-524010b6a352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Oct 29 23:10:03.727747 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:03.725760 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:03.727747 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:03.726116 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 29 23:10:03.727747 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:03.726396 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 29 23:10:03.766316 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:03.766273 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:03.819095 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:03.818928 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:10:03.820887 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:03.820608 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:10:03.856909 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:03.854802 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:03.976227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:03.973976 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:04.329304 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:04.328955 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:04.359359 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:04.359315 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:04.501055 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:04.501011 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:04.509808 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:04.502848 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:04.589282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:04.538787 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e8612f6-84jxm"] Oct 29 23:10:04.589282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:04.578421 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e8612f6-84jxm"] Oct 29 23:10:04.606333 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:04.606273 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 23:10:04.608450 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:04.608406 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:04.662442 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:04.654270 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 23:10:04.696779 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:04.696735 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 29 23:10:04.719229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:04.698705 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 23:10:04.730370 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:04.730327 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:04.789273 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:04.789234 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 29 23:10:04.800235 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:04.790913 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 23:10:04.800719 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:04.800651 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 23:10:04.852288 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:04.851294 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:10:04.852288 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:04.851362 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" podUID="137c7cf5-e922-49d1-acbc-fc0b13899da5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:04.871846 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:04.871692 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:05.158030 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.133876 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 23:10:05.158030 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.134336 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 23:10:05.158030 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.134901 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded" Oct 29 23:10:05.160269 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.159838 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 29 23:10:05.245236 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.234297 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:10:05.245236 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.234362 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" podUID="62c0101f-490c-470e-b945-e3bb7c86d2c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:05.327228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.300053 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:10:05.327228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.301432 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:10:05.327228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.304530 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:10:05.369227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.343429 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:05.373162 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.369911 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 23:10:05.373162 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.371927 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 29 23:10:05.373162 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.372310 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 23:10:05.445072 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.444974 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 23:10:05.552246 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.547279 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 23:10:05.552588 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.552554 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:05.575340 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.575275 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 23:10:05.583298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.575384 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:05.583298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.576792 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:10:05.583298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.580746 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" podUID="041cb60f-5e7c-47d7-8e83-cce35d4c9529" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:05.583298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.580845 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 23:10:05.698286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.697450 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:05.777295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.764001 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:10:05.777295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.764069 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" podUID="c0da6fd9-efd5-4587-b764-524010b6a352" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:05.777295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.775006 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:05.819234 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.800267 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:05.946064 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.946025 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 23:10:05.967767 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.967659 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 23:10:05.970914 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.970894 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 23:10:05.985800 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.980070 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 23:10:05.997670 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:05.997639 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 23:10:06.140916 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:06.140880 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:10:06.161928 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:06.161898 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:10:06.356554 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:06.356427 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="448249df-154f-42be-9621-28d77dc52679" path="/var/lib/kubelet/pods/448249df-154f-42be-9621-28d77dc52679/volumes" Oct 29 23:10:06.357459 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:06.357430 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:10:13.206688 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.206649 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts"] Oct 29 23:10:13.212346 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.212323 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.238285 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.238256 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts"] Oct 29 23:10:13.243229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.243187 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-a9ced3-openshift-default-dockercfg-mhf8n\"" Oct 29 23:10:13.244118 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.244094 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09716f15-687a-47d5-960a-19beab856a8b-istio-token\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.244228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.244152 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-istio-data\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.244352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.244333 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-workload-certs\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.244432 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.244372 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-465xj\" (UniqueName: \"kubernetes.io/projected/09716f15-687a-47d5-960a-19beab856a8b-kube-api-access-465xj\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.244490 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.244457 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-credential-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.244570 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.244522 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09716f15-687a-47d5-960a-19beab856a8b-istiod-ca-cert\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.244751 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.244667 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-istio-envoy\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.244751 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.244741 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-workload-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.244861 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.244770 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09716f15-687a-47d5-960a-19beab856a8b-istio-podinfo\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.346726 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.346697 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09716f15-687a-47d5-960a-19beab856a8b-istio-podinfo\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.346932 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.346809 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09716f15-687a-47d5-960a-19beab856a8b-istio-token\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.346932 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.346841 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-istio-data\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.346932 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.346879 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-workload-certs\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.346932 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.346898 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-465xj\" (UniqueName: \"kubernetes.io/projected/09716f15-687a-47d5-960a-19beab856a8b-kube-api-access-465xj\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.347172 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.346949 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-credential-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.347172 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.346994 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09716f15-687a-47d5-960a-19beab856a8b-istiod-ca-cert\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.347172 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.347087 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-istio-envoy\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.347172 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.347166 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-workload-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.347422 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.347374 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-istio-data\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.347491 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.347443 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-workload-certs\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.347554 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.347485 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-credential-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.347775 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.347668 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-workload-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.347882 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.347858 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09716f15-687a-47d5-960a-19beab856a8b-istiod-ca-cert\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.349499 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.349471 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09716f15-687a-47d5-960a-19beab856a8b-istio-podinfo\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.349688 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.349666 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-istio-envoy\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.371403 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.371382 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09716f15-687a-47d5-960a-19beab856a8b-istio-token\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.371584 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.371565 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-465xj\" (UniqueName: \"kubernetes.io/projected/09716f15-687a-47d5-960a-19beab856a8b-kube-api-access-465xj\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.527254 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.527150 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:13.881337 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:10:13.881291 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09716f15_687a_47d5_960a_19beab856a8b.slice/crio-b36198b910d4a3b1a1314e843f79043a1ff56c42da11fc38f78084fe1b9ac3f6 WatchSource:0}: Error finding container b36198b910d4a3b1a1314e843f79043a1ff56c42da11fc38f78084fe1b9ac3f6: Status 404 returned error can't find the container with id b36198b910d4a3b1a1314e843f79043a1ff56c42da11fc38f78084fe1b9ac3f6 Oct 29 23:10:13.883599 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.883434 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts"] Oct 29 23:10:13.884931 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.884876 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:10:13.885056 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.884974 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:10:13.885056 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:13.885022 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:10:14.197017 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:14.196915 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" event={"ID":"09716f15-687a-47d5-960a-19beab856a8b","Type":"ContainerStarted","Data":"9b2658faa70a9462a4b71645c08200925d12dffb7af52a05224af2663661cba7"} Oct 29 23:10:14.197017 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:14.196980 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" event={"ID":"09716f15-687a-47d5-960a-19beab856a8b","Type":"ContainerStarted","Data":"b36198b910d4a3b1a1314e843f79043a1ff56c42da11fc38f78084fe1b9ac3f6"} Oct 29 23:10:14.251317 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:14.251265 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" podStartSLOduration=1.251250186 podStartE2EDuration="1.251250186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:14.244377361 +0000 UTC m=+1668.184925274" watchObservedRunningTime="2026-10-29 23:10:14.251250186 +0000 UTC m=+1668.191798095" Oct 29 23:10:14.528637 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:14.527802 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:14.530773 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:14.530730 4821 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 29 23:10:14.530974 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:14.530798 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" podUID="09716f15-687a-47d5-960a-19beab856a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 29 23:10:14.873374 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:14.873335 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb"] Oct 29 23:10:14.873654 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:14.873590 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" podUID="b1fb295f-b287-441f-a057-efaf16a8c8cc" containerName="istio-proxy" containerID="cri-o://0c12224e7da4e3f0c66a9c4d8dd198c68acfd2cfd73706e843d500fffb69b16e" gracePeriod=30 Oct 29 23:10:15.539711 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:15.539668 4821 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 29 23:10:15.540099 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:15.539742 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" podUID="09716f15-687a-47d5-960a-19beab856a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 29 23:10:16.537228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.536402 4821 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 29 23:10:16.537228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.536460 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" podUID="09716f15-687a-47d5-960a-19beab856a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 29 23:10:16.683244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.683191 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw"] Oct 29 23:10:16.688037 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.688016 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.711833 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.711809 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw"] Oct 29 23:10:16.733226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.729734 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-af237d-openshift-default-dockercfg-9g5k4\"" Oct 29 23:10:16.806540 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.806448 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-workload-certs\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.806540 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.806522 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-credential-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.806733 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.806634 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-envoy\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.806733 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.806673 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-workload-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.806836 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.806743 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-data\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.806938 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.806918 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-podinfo\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.806998 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.806972 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-token\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.807104 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.807089 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9mhm\" (UniqueName: \"kubernetes.io/projected/77d46565-e43f-44a8-8d78-4417e7016c5d-kube-api-access-s9mhm\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.807249 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.807226 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d46565-e43f-44a8-8d78-4417e7016c5d-istiod-ca-cert\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.913503 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.913457 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-podinfo\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.923641 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.915529 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-token\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.924015 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.923991 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9mhm\" (UniqueName: \"kubernetes.io/projected/77d46565-e43f-44a8-8d78-4417e7016c5d-kube-api-access-s9mhm\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.928118 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.924543 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d46565-e43f-44a8-8d78-4417e7016c5d-istiod-ca-cert\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.929053 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.929027 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-workload-certs\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.939725 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.925993 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d46565-e43f-44a8-8d78-4417e7016c5d-istiod-ca-cert\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.940012 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.925259 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-podinfo\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.944844 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.929524 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-workload-certs\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.945095 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.945022 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-credential-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.945548 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.945528 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-envoy\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.952927 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.952900 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-workload-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.957662 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.957598 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-workload-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.958177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.958130 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-data\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.983259 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.953195 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-data\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.983259 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.971641 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-credential-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.992770 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.992735 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9mhm\" (UniqueName: \"kubernetes.io/projected/77d46565-e43f-44a8-8d78-4417e7016c5d-kube-api-access-s9mhm\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.995678 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.993513 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-envoy\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:16.996354 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:16.996309 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-token\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:17.005767 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:17.005529 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:17.468428 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:10:17.468396 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d46565_e43f_44a8_8d78_4417e7016c5d.slice/crio-abd652882197937d72079ab98ac6fe6da75ad2fce09525ba205e8d46f5ad10c9 WatchSource:0}: Error finding container abd652882197937d72079ab98ac6fe6da75ad2fce09525ba205e8d46f5ad10c9: Status 404 returned error can't find the container with id abd652882197937d72079ab98ac6fe6da75ad2fce09525ba205e8d46f5ad10c9 Oct 29 23:10:17.468590 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:17.468564 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw"] Oct 29 23:10:17.471465 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:17.471425 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:10:17.471580 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:17.471534 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:10:17.471650 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:17.471586 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:10:17.542256 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:17.540887 4821 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 29 23:10:17.542256 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:17.540949 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" podUID="09716f15-687a-47d5-960a-19beab856a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 29 23:10:18.286485 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:18.286439 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" event={"ID":"77d46565-e43f-44a8-8d78-4417e7016c5d","Type":"ContainerStarted","Data":"2147d171319ef1ed54a31f87fa4113cc09f04fdff294d37bee328d4640bf3e07"} Oct 29 23:10:18.287059 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:18.286502 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" event={"ID":"77d46565-e43f-44a8-8d78-4417e7016c5d","Type":"ContainerStarted","Data":"abd652882197937d72079ab98ac6fe6da75ad2fce09525ba205e8d46f5ad10c9"} Oct 29 23:10:18.370228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:18.369653 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" podStartSLOduration=2.369633647 podStartE2EDuration="2.369633647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:18.36789939 +0000 UTC m=+1672.308447298" watchObservedRunningTime="2026-10-29 23:10:18.369633647 +0000 UTC m=+1672.310181557" Oct 29 23:10:18.536227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:18.535878 4821 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 29 23:10:18.536227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:18.535943 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" podUID="09716f15-687a-47d5-960a-19beab856a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 29 23:10:19.013382 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:19.013332 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:19.022227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:19.018310 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 29 23:10:19.022227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:19.018367 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" podUID="77d46565-e43f-44a8-8d78-4417e7016c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 29 23:10:19.667224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:19.553159 4821 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body= Oct 29 23:10:19.709585 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:19.709463 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" podUID="09716f15-687a-47d5-960a-19beab856a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" Oct 29 23:10:20.049536 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.016628 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 29 23:10:20.049536 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.016687 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" podUID="77d46565-e43f-44a8-8d78-4417e7016c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 29 23:10:20.317586 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.316317 4821 generic.go:356] "Generic (PLEG): container finished" podID="b1fb295f-b287-441f-a057-efaf16a8c8cc" containerID="0c12224e7da4e3f0c66a9c4d8dd198c68acfd2cfd73706e843d500fffb69b16e" exitCode=0 Oct 29 23:10:20.317586 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.317461 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" event={"ID":"b1fb295f-b287-441f-a057-efaf16a8c8cc","Type":"ContainerDied","Data":"0c12224e7da4e3f0c66a9c4d8dd198c68acfd2cfd73706e843d500fffb69b16e"} Oct 29 23:10:20.423228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.422953 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 23:10:20.608352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.602053 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1fb295f-b287-441f-a057-efaf16a8c8cc-kube-api-access-vrf9c\" (UniqueName: \"kubernetes.io/projected/b1fb295f-b287-441f-a057-efaf16a8c8cc-kube-api-access-vrf9c\") pod \"b1fb295f-b287-441f-a057-efaf16a8c8cc\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " Oct 29 23:10:20.608352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.602104 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-workload-socket\") pod \"b1fb295f-b287-441f-a057-efaf16a8c8cc\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " Oct 29 23:10:20.608352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.602148 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-token\" (UniqueName: \"kubernetes.io/projected/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-token\") pod \"b1fb295f-b287-441f-a057-efaf16a8c8cc\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " Oct 29 23:10:20.608352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.602230 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-credential-socket\") pod \"b1fb295f-b287-441f-a057-efaf16a8c8cc\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " Oct 29 23:10:20.608352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.602336 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-envoy\") pod \"b1fb295f-b287-441f-a057-efaf16a8c8cc\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " Oct 29 23:10:20.608352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.602467 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1fb295f-b287-441f-a057-efaf16a8c8cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1fb295f-b287-441f-a057-efaf16a8c8cc-istiod-ca-cert\") pod \"b1fb295f-b287-441f-a057-efaf16a8c8cc\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " Oct 29 23:10:20.608352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.602504 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-podinfo\") pod \"b1fb295f-b287-441f-a057-efaf16a8c8cc\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " Oct 29 23:10:20.608352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.602625 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-workload-certs\") pod \"b1fb295f-b287-441f-a057-efaf16a8c8cc\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " Oct 29 23:10:20.608352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.602674 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-data\") pod \"b1fb295f-b287-441f-a057-efaf16a8c8cc\" (UID: \"b1fb295f-b287-441f-a057-efaf16a8c8cc\") " Oct 29 23:10:20.608352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.604084 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-data" pod "b1fb295f-b287-441f-a057-efaf16a8c8cc" (UID: "b1fb295f-b287-441f-a057-efaf16a8c8cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:20.608352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.604139 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-credential-socket" pod "b1fb295f-b287-441f-a057-efaf16a8c8cc" (UID: "b1fb295f-b287-441f-a057-efaf16a8c8cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:20.608352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.604462 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1fb295f-b287-441f-a057-efaf16a8c8cc-istiod-ca-cert" pod "b1fb295f-b287-441f-a057-efaf16a8c8cc" (UID: "b1fb295f-b287-441f-a057-efaf16a8c8cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:10:20.608352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.604697 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-workload-certs" pod "b1fb295f-b287-441f-a057-efaf16a8c8cc" (UID: "b1fb295f-b287-441f-a057-efaf16a8c8cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:20.608352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.604929 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-workload-socket" pod "b1fb295f-b287-441f-a057-efaf16a8c8cc" (UID: "b1fb295f-b287-441f-a057-efaf16a8c8cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:20.613400 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.613368 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-token" pod "b1fb295f-b287-441f-a057-efaf16a8c8cc" (UID: "b1fb295f-b287-441f-a057-efaf16a8c8cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:20.616447 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.616391 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1fb295f-b287-441f-a057-efaf16a8c8cc-kube-api-access-vrf9c" pod "b1fb295f-b287-441f-a057-efaf16a8c8cc" (UID: "b1fb295f-b287-441f-a057-efaf16a8c8cc"). InnerVolumeSpecName "kube-api-access-vrf9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:10:20.616665 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.616634 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-envoy" pod "b1fb295f-b287-441f-a057-efaf16a8c8cc" (UID: "b1fb295f-b287-441f-a057-efaf16a8c8cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:10:20.617025 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.617001 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-podinfo" pod "b1fb295f-b287-441f-a057-efaf16a8c8cc" (UID: "b1fb295f-b287-441f-a057-efaf16a8c8cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:10:20.705232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.705162 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:10:20.705232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.705214 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1fb295f-b287-441f-a057-efaf16a8c8cc-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:10:20.705232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.705230 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:10:20.705740 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.705243 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:10:20.705740 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.705256 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:10:20.705740 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.705269 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrf9c\" (UniqueName: \"kubernetes.io/projected/b1fb295f-b287-441f-a057-efaf16a8c8cc-kube-api-access-vrf9c\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:10:20.705740 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.705286 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:10:20.705740 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.705299 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1fb295f-b287-441f-a057-efaf16a8c8cc-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:10:20.705740 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.705313 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fb295f-b287-441f-a057-efaf16a8c8cc-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:10:20.886474 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.886371 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:20.887023 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.886943 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:20.887979 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:20.887959 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:10:21.046212 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:21.046157 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 29 23:10:21.046519 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:21.046493 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" podUID="77d46565-e43f-44a8-8d78-4417e7016c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 29 23:10:21.345346 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:21.345249 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" event={"ID":"b1fb295f-b287-441f-a057-efaf16a8c8cc","Type":"ContainerDied","Data":"a558cde18403b9b3309a8c0cf2273d13161438014f514dd52314847245462238"} Oct 29 23:10:21.345346 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:21.345297 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb" Oct 29 23:10:21.345346 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:21.345321 4821 scope.go:122] "RemoveContainer" containerID="0c12224e7da4e3f0c66a9c4d8dd198c68acfd2cfd73706e843d500fffb69b16e" Oct 29 23:10:21.405952 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:21.405917 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb"] Oct 29 23:10:21.449225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:21.446028 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-2hddb"] Oct 29 23:10:22.013886 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:22.013842 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 29 23:10:22.014406 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:22.013893 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" podUID="77d46565-e43f-44a8-8d78-4417e7016c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 29 23:10:22.172009 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:22.171972 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1fb295f-b287-441f-a057-efaf16a8c8cc" path="/var/lib/kubelet/pods/b1fb295f-b287-441f-a057-efaf16a8c8cc/volumes" Oct 29 23:10:22.902600 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:22.902566 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp"] Oct 29 23:10:22.903161 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:22.903146 4821 cpu_manager.go:414] "containerMap: removing container" podUID="b1fb295f-b287-441f-a057-efaf16a8c8cc" containerName="istio-proxy" Oct 29 23:10:22.903161 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:22.903160 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1fb295f-b287-441f-a057-efaf16a8c8cc" containerName="istio-proxy" Oct 29 23:10:22.903304 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:22.903292 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1fb295f-b287-441f-a057-efaf16a8c8cc" containerName="istio-proxy" Oct 29 23:10:22.908380 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:22.908359 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:22.942224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:22.936686 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-32717a-openshift-default-dockercfg-4th7d\"" Oct 29 23:10:22.955154 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:22.955124 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp"] Oct 29 23:10:22.981954 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:22.981922 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-token\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:22.982146 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:22.981968 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-credential-socket\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:22.982146 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:22.982056 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-data\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:22.982146 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:22.982095 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-envoy\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:22.982340 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:22.982176 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-workload-certs\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:22.982340 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:22.982227 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp8z5\" (UniqueName: \"kubernetes.io/projected/df9f4a39-1043-449e-b5b3-b57c82b618b3-kube-api-access-vp8z5\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:22.982340 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:22.982252 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df9f4a39-1043-449e-b5b3-b57c82b618b3-istiod-ca-cert\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:22.982340 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:22.982270 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-workload-socket\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:22.982340 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:22.982287 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-podinfo\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:23.013414 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.013383 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 29 23:10:23.013596 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.013438 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" podUID="77d46565-e43f-44a8-8d78-4417e7016c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 29 23:10:23.083524 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.083488 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-token\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:23.084002 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.083547 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-credential-socket\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:23.084002 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.083654 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-data\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:23.084002 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.083708 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-envoy\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:23.084002 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.083808 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-workload-certs\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:23.084002 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.083849 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp8z5\" (UniqueName: \"kubernetes.io/projected/df9f4a39-1043-449e-b5b3-b57c82b618b3-kube-api-access-vp8z5\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:23.084002 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.083879 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df9f4a39-1043-449e-b5b3-b57c82b618b3-istiod-ca-cert\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:23.084002 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.083898 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-workload-socket\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:23.084002 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.083923 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-podinfo\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:23.084002 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.083973 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-credential-socket\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:23.084511 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.084178 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-workload-certs\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:23.084511 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.084294 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-data\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:23.084511 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.084355 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-workload-socket\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:23.084847 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.084825 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df9f4a39-1043-449e-b5b3-b57c82b618b3-istiod-ca-cert\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:23.086218 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.086182 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-envoy\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:23.086615 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.086598 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-podinfo\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:23.109425 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.109400 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-token\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:23.109647 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.109628 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp8z5\" (UniqueName: \"kubernetes.io/projected/df9f4a39-1043-449e-b5b3-b57c82b618b3-kube-api-access-vp8z5\") pod \"e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:23.251860 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.251767 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:23.601083 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:10:23.601050 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9f4a39_1043_449e_b5b3_b57c82b618b3.slice/crio-23cf309f5201487e83a5893c7932e749b8144cd153be1599fd37b583640d0e51 WatchSource:0}: Error finding container 23cf309f5201487e83a5893c7932e749b8144cd153be1599fd37b583640d0e51: Status 404 returned error can't find the container with id 23cf309f5201487e83a5893c7932e749b8144cd153be1599fd37b583640d0e51 Oct 29 23:10:23.603191 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.603163 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp"] Oct 29 23:10:23.604086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.604055 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:10:23.604146 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.604123 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:10:23.604183 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:23.604155 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:10:23.890369 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:10:23.890320 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:10:24.077605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:24.077565 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:24.078852 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:24.078821 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:24.083409 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:24.083362 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:10:24.389724 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:24.389681 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" event={"ID":"df9f4a39-1043-449e-b5b3-b57c82b618b3","Type":"ContainerStarted","Data":"c4948e75f358f5cd5e3cbf70b52b02fa3804855bdb9c464547466161a0df51ef"} Oct 29 23:10:24.390470 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:24.389751 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" event={"ID":"df9f4a39-1043-449e-b5b3-b57c82b618b3","Type":"ContainerStarted","Data":"23cf309f5201487e83a5893c7932e749b8144cd153be1599fd37b583640d0e51"} Oct 29 23:10:24.453072 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:24.453012 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" podStartSLOduration=2.452996832 podStartE2EDuration="2.452996832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:24.447365787 +0000 UTC m=+1678.387913709" watchObservedRunningTime="2026-10-29 23:10:24.452996832 +0000 UTC m=+1678.393544741" Oct 29 23:10:25.251964 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:25.251928 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:25.253451 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:25.253417 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Oct 29 23:10:25.253607 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:25.253478 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" podUID="df9f4a39-1043-449e-b5b3-b57c82b618b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Oct 29 23:10:26.258614 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:26.258571 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Oct 29 23:10:26.258989 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:26.258651 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" podUID="df9f4a39-1043-449e-b5b3-b57c82b618b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Oct 29 23:10:27.252960 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:27.252924 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Oct 29 23:10:27.253169 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:27.252980 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" podUID="df9f4a39-1043-449e-b5b3-b57c82b618b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Oct 29 23:10:28.253142 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:28.252617 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Oct 29 23:10:28.253142 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:28.252691 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" podUID="df9f4a39-1043-449e-b5b3-b57c82b618b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Oct 29 23:10:28.894780 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:28.894749 4821 scope.go:122] "RemoveContainer" containerID="6e43f16535746b5e36916ee3cc736adc6917561e04fd094fb24df0228cfb6394" Oct 29 23:10:28.906007 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:28.905983 4821 scope.go:122] "RemoveContainer" containerID="d9374905b10875f93d83af569ebe875f6b7fc25d388b036f2b94487ec018f618" Oct 29 23:10:28.958239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:28.957880 4821 scope.go:122] "RemoveContainer" containerID="916359ebc8901531efd95c4bf5e188cfd3387f86674a4fb1782faad55576b1ef" Oct 29 23:10:29.253032 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:29.252942 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Oct 29 23:10:29.253032 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:29.253008 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" podUID="df9f4a39-1043-449e-b5b3-b57c82b618b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Oct 29 23:10:30.256975 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:30.256941 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:30.257612 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:30.257441 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:30.258322 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:30.258299 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:10:35.070996 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:35.070955 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp"] Oct 29 23:10:35.077071 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:35.077047 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp" Oct 29 23:10:35.101687 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:35.101650 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp"] Oct 29 23:10:35.106977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:35.106950 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-67363b-serving-cert\"" Oct 29 23:10:35.172396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:35.172365 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bab23b35-8838-415c-90bf-aebf41448b58-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp\" (UID: \"bab23b35-8838-415c-90bf-aebf41448b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp" Oct 29 23:10:35.172573 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:35.172496 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hgh7\" (UniqueName: \"kubernetes.io/projected/bab23b35-8838-415c-90bf-aebf41448b58-kube-api-access-9hgh7\") pod \"maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp\" (UID: \"bab23b35-8838-415c-90bf-aebf41448b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp" Oct 29 23:10:35.172573 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:35.172552 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab23b35-8838-415c-90bf-aebf41448b58-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp\" (UID: \"bab23b35-8838-415c-90bf-aebf41448b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp" Oct 29 23:10:35.172698 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:35.172671 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bab23b35-8838-415c-90bf-aebf41448b58-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp\" (UID: \"bab23b35-8838-415c-90bf-aebf41448b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp" Oct 29 23:10:35.274667 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:35.274629 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bab23b35-8838-415c-90bf-aebf41448b58-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp\" (UID: \"bab23b35-8838-415c-90bf-aebf41448b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp" Oct 29 23:10:35.274854 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:35.274811 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bab23b35-8838-415c-90bf-aebf41448b58-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp\" (UID: \"bab23b35-8838-415c-90bf-aebf41448b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp" Oct 29 23:10:35.275343 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:35.274991 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hgh7\" (UniqueName: \"kubernetes.io/projected/bab23b35-8838-415c-90bf-aebf41448b58-kube-api-access-9hgh7\") pod \"maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp\" (UID: \"bab23b35-8838-415c-90bf-aebf41448b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp" Oct 29 23:10:35.275343 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:35.275101 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab23b35-8838-415c-90bf-aebf41448b58-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp\" (UID: \"bab23b35-8838-415c-90bf-aebf41448b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp" Oct 29 23:10:35.275982 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:35.275895 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab23b35-8838-415c-90bf-aebf41448b58-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp\" (UID: \"bab23b35-8838-415c-90bf-aebf41448b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp" Oct 29 23:10:35.277687 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:35.277660 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bab23b35-8838-415c-90bf-aebf41448b58-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp\" (UID: \"bab23b35-8838-415c-90bf-aebf41448b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp" Oct 29 23:10:35.277784 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:35.277736 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bab23b35-8838-415c-90bf-aebf41448b58-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp\" (UID: \"bab23b35-8838-415c-90bf-aebf41448b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp" Oct 29 23:10:35.300286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:35.300255 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hgh7\" (UniqueName: \"kubernetes.io/projected/bab23b35-8838-415c-90bf-aebf41448b58-kube-api-access-9hgh7\") pod \"maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp\" (UID: \"bab23b35-8838-415c-90bf-aebf41448b58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp" Oct 29 23:10:35.392908 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:35.392542 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp" Oct 29 23:10:35.766856 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:35.766809 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp"] Oct 29 23:10:36.743842 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:36.743728 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp" event={"ID":"bab23b35-8838-415c-90bf-aebf41448b58","Type":"ContainerStarted","Data":"b33053c0bc1933c80d4e767a974d2bd8dda3e82ca6e73778c81f61c5cd218838"} Oct 29 23:10:36.743842 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:36.743829 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp" event={"ID":"bab23b35-8838-415c-90bf-aebf41448b58","Type":"ContainerStarted","Data":"2651901ebcbb6cd2e60462e188a8666be1ef56ff3d8b05a7dd4b7acf1c14ba19"} Oct 29 23:10:36.744399 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:36.743881 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp" Oct 29 23:10:36.792445 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:36.792366 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp" podStartSLOduration=1.125010922 podStartE2EDuration="1.792352479s" totalImagesPullingTime="667.341557ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:36.787828099 +0000 UTC m=+1690.728376102" watchObservedRunningTime="2026-10-29 23:10:36.792352479 +0000 UTC m=+1690.732900432" Oct 29 23:10:39.711224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:39.710483 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 29 23:10:39.785225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:39.784393 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:42.199175 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:42.199145 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq"] Oct 29 23:10:42.203045 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:42.203028 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq" Oct 29 23:10:42.227282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:42.227253 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq"] Oct 29 23:10:42.227666 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:42.227648 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-3e5b21-serving-cert\"" Oct 29 23:10:42.245681 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:42.245649 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v49qm\" (UniqueName: \"kubernetes.io/projected/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-kube-api-access-v49qm\") pod \"maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq\" (UID: \"e5c6effb-9b40-4c4f-aaa7-25bb59ce5469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq" Oct 29 23:10:42.245827 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:42.245729 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq\" (UID: \"e5c6effb-9b40-4c4f-aaa7-25bb59ce5469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq" Oct 29 23:10:42.245875 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:42.245849 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq\" (UID: \"e5c6effb-9b40-4c4f-aaa7-25bb59ce5469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq" Oct 29 23:10:42.245935 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:42.245920 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq\" (UID: \"e5c6effb-9b40-4c4f-aaa7-25bb59ce5469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq" Oct 29 23:10:42.347550 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:42.347516 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v49qm\" (UniqueName: \"kubernetes.io/projected/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-kube-api-access-v49qm\") pod \"maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq\" (UID: \"e5c6effb-9b40-4c4f-aaa7-25bb59ce5469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq" Oct 29 23:10:42.347732 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:42.347601 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq\" (UID: \"e5c6effb-9b40-4c4f-aaa7-25bb59ce5469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq" Oct 29 23:10:42.347732 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:42.347655 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq\" (UID: \"e5c6effb-9b40-4c4f-aaa7-25bb59ce5469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq" Oct 29 23:10:42.347732 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:42.347715 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq\" (UID: \"e5c6effb-9b40-4c4f-aaa7-25bb59ce5469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq" Oct 29 23:10:42.348424 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:42.348360 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq\" (UID: \"e5c6effb-9b40-4c4f-aaa7-25bb59ce5469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq" Oct 29 23:10:42.350333 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:42.350305 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq\" (UID: \"e5c6effb-9b40-4c4f-aaa7-25bb59ce5469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq" Oct 29 23:10:42.350465 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:42.350441 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq\" (UID: \"e5c6effb-9b40-4c4f-aaa7-25bb59ce5469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq" Oct 29 23:10:42.375222 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:42.375167 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v49qm\" (UniqueName: \"kubernetes.io/projected/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-kube-api-access-v49qm\") pod \"maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq\" (UID: \"e5c6effb-9b40-4c4f-aaa7-25bb59ce5469\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq" Oct 29 23:10:42.517899 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:42.517817 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq" Oct 29 23:10:42.758422 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:42.758395 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp" Oct 29 23:10:42.866482 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:10:42.866450 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c6effb_9b40_4c4f_aaa7_25bb59ce5469.slice/crio-6153977aaea403cb3e07f8a6ee7ef104a20cfa68207da8a2d3884ab2cfa0fe9f WatchSource:0}: Error finding container 6153977aaea403cb3e07f8a6ee7ef104a20cfa68207da8a2d3884ab2cfa0fe9f: Status 404 returned error can't find the container with id 6153977aaea403cb3e07f8a6ee7ef104a20cfa68207da8a2d3884ab2cfa0fe9f Oct 29 23:10:42.868810 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:42.868787 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq"] Oct 29 23:10:43.840980 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:43.840940 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq" event={"ID":"e5c6effb-9b40-4c4f-aaa7-25bb59ce5469","Type":"ContainerStarted","Data":"5444c7447557639fe9d79044cf57751ff48bf3dfb52b8e0e6e6bfae2524c2758"} Oct 29 23:10:43.841445 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:43.841007 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq" event={"ID":"e5c6effb-9b40-4c4f-aaa7-25bb59ce5469","Type":"ContainerStarted","Data":"6153977aaea403cb3e07f8a6ee7ef104a20cfa68207da8a2d3884ab2cfa0fe9f"} Oct 29 23:10:43.841445 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:43.841038 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq" Oct 29 23:10:43.902665 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:43.902616 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq" podStartSLOduration=1.491908158 podStartE2EDuration="1.902598912s" totalImagesPullingTime="410.690754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:43.898117416 +0000 UTC m=+1697.838665327" watchObservedRunningTime="2026-10-29 23:10:43.902598912 +0000 UTC m=+1697.843146818" Oct 29 23:10:49.854505 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:49.854473 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq" Oct 29 23:10:50.796580 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:50.796546 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj"] Oct 29 23:10:50.801486 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:50.801465 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" Oct 29 23:10:50.824253 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:50.824021 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj"] Oct 29 23:10:50.827396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:50.827022 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-755453-serving-cert\"" Oct 29 23:10:50.901730 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:50.901693 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d192d0e-6dcf-4979-a503-6047643e055e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj\" (UID: \"8d192d0e-6dcf-4979-a503-6047643e055e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" Oct 29 23:10:50.902125 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:50.901740 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85d44\" (UniqueName: \"kubernetes.io/projected/8d192d0e-6dcf-4979-a503-6047643e055e-kube-api-access-85d44\") pod \"maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj\" (UID: \"8d192d0e-6dcf-4979-a503-6047643e055e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" Oct 29 23:10:50.902125 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:50.901988 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d192d0e-6dcf-4979-a503-6047643e055e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj\" (UID: \"8d192d0e-6dcf-4979-a503-6047643e055e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" Oct 29 23:10:50.902125 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:50.902022 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d192d0e-6dcf-4979-a503-6047643e055e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj\" (UID: \"8d192d0e-6dcf-4979-a503-6047643e055e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" Oct 29 23:10:51.004943 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:51.004452 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d192d0e-6dcf-4979-a503-6047643e055e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj\" (UID: \"8d192d0e-6dcf-4979-a503-6047643e055e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" Oct 29 23:10:51.004943 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:51.004522 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d192d0e-6dcf-4979-a503-6047643e055e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj\" (UID: \"8d192d0e-6dcf-4979-a503-6047643e055e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" Oct 29 23:10:51.004943 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:51.004727 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d192d0e-6dcf-4979-a503-6047643e055e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj\" (UID: \"8d192d0e-6dcf-4979-a503-6047643e055e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" Oct 29 23:10:51.004943 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:51.004765 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85d44\" (UniqueName: \"kubernetes.io/projected/8d192d0e-6dcf-4979-a503-6047643e055e-kube-api-access-85d44\") pod \"maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj\" (UID: \"8d192d0e-6dcf-4979-a503-6047643e055e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" Oct 29 23:10:51.012220 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:10:51.009037 4821 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-serving-cert: secret "maas-api-e2e-shared-b-w3-755453-serving-cert" not found Oct 29 23:10:51.012220 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:10:51.009122 4821 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d192d0e-6dcf-4979-a503-6047643e055e-maas-api-tls podName:8d192d0e-6dcf-4979-a503-6047643e055e nodeName:}" failed. No retries permitted until 2026-10-29 23:10:51.50909968 +0000 UTC m=+1705.449647586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8d192d0e-6dcf-4979-a503-6047643e055e-maas-api-tls") pod "maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" (UID: "8d192d0e-6dcf-4979-a503-6047643e055e") : secret "maas-api-e2e-shared-b-w3-755453-serving-cert" not found Oct 29 23:10:51.012220 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:51.010155 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d192d0e-6dcf-4979-a503-6047643e055e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj\" (UID: \"8d192d0e-6dcf-4979-a503-6047643e055e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" Oct 29 23:10:51.021290 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:51.021266 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d192d0e-6dcf-4979-a503-6047643e055e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj\" (UID: \"8d192d0e-6dcf-4979-a503-6047643e055e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" Oct 29 23:10:51.039183 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:51.039153 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85d44\" (UniqueName: \"kubernetes.io/projected/8d192d0e-6dcf-4979-a503-6047643e055e-kube-api-access-85d44\") pod \"maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj\" (UID: \"8d192d0e-6dcf-4979-a503-6047643e055e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" Oct 29 23:10:51.518427 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:51.518395 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d192d0e-6dcf-4979-a503-6047643e055e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj\" (UID: \"8d192d0e-6dcf-4979-a503-6047643e055e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" Oct 29 23:10:51.523163 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:51.521846 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d192d0e-6dcf-4979-a503-6047643e055e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj\" (UID: \"8d192d0e-6dcf-4979-a503-6047643e055e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" Oct 29 23:10:51.716887 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:51.716853 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" Oct 29 23:10:52.102895 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:10:52.102729 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d192d0e_6dcf_4979_a503_6047643e055e.slice/crio-90b315d661564fb89c31e65a328358abcfc34ff417ec8b7709c2110feaa0ea99 WatchSource:0}: Error finding container 90b315d661564fb89c31e65a328358abcfc34ff417ec8b7709c2110feaa0ea99: Status 404 returned error can't find the container with id 90b315d661564fb89c31e65a328358abcfc34ff417ec8b7709c2110feaa0ea99 Oct 29 23:10:52.114511 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:52.114480 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj"] Oct 29 23:10:52.923789 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:52.923744 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" event={"ID":"8d192d0e-6dcf-4979-a503-6047643e055e","Type":"ContainerStarted","Data":"5e5c8fe5676eed629d233f5fab6040542f0f9a013d5086a87cf512732f67096c"} Oct 29 23:10:52.923972 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:52.923818 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" event={"ID":"8d192d0e-6dcf-4979-a503-6047643e055e","Type":"ContainerStarted","Data":"90b315d661564fb89c31e65a328358abcfc34ff417ec8b7709c2110feaa0ea99"} Oct 29 23:10:52.923972 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:52.923848 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" Oct 29 23:10:53.887486 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:10:53.887438 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:10:58.356224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.356040 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" podStartSLOduration=8.065909514 podStartE2EDuration="8.35601884s" totalImagesPullingTime="290.109326ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:10:52.992520235 +0000 UTC m=+1706.933068146" watchObservedRunningTime="2026-10-29 23:10:58.35601884 +0000 UTC m=+1712.296566750" Oct 29 23:10:58.360258 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.357776 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv"] Oct 29 23:10:58.371222 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.371155 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.393237 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.393182 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv"] Oct 29 23:10:58.433767 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.433727 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-podinfo\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.433958 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.433932 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-workload-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.434028 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.433999 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-envoy\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.434151 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.434127 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870e234a-d4fd-4877-a458-d8bbfd13b010-istiod-ca-cert\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.434243 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.434185 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4csm\" (UniqueName: \"kubernetes.io/projected/870e234a-d4fd-4877-a458-d8bbfd13b010-kube-api-access-k4csm\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.434350 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.434328 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-credential-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.434432 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.434414 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-workload-certs\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.434550 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.434535 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-token\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.434609 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.434595 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-data\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.535774 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.535740 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-token\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.535987 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.535813 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-data\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.535987 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.535847 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-podinfo\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.535987 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.535979 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-workload-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.536155 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.536043 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-envoy\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.536155 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.536103 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870e234a-d4fd-4877-a458-d8bbfd13b010-istiod-ca-cert\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.536298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.536153 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4csm\" (UniqueName: \"kubernetes.io/projected/870e234a-d4fd-4877-a458-d8bbfd13b010-kube-api-access-k4csm\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.536298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.536229 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-credential-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.536400 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.536290 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-data\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.536400 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.536298 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-workload-certs\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.536519 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.536456 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-workload-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.536764 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.536732 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-credential-socket\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.536764 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.536759 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-workload-certs\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.537098 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.537079 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870e234a-d4fd-4877-a458-d8bbfd13b010-istiod-ca-cert\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.538884 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.538858 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-envoy\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.539072 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.539057 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-podinfo\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.562037 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.561997 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-token\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.562269 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.562246 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4csm\" (UniqueName: \"kubernetes.io/projected/870e234a-d4fd-4877-a458-d8bbfd13b010-kube-api-access-k4csm\") pod \"e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.690330 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.690294 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:10:58.937550 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:58.937522 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" Oct 29 23:10:59.051220 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:59.051083 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv"] Oct 29 23:10:59.053805 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:59.053763 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:10:59.053919 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:59.053849 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:10:59.053919 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:59.053887 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:10:59.989722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:59.989680 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" event={"ID":"870e234a-d4fd-4877-a458-d8bbfd13b010","Type":"ContainerStarted","Data":"750e7bd23c5129f13d2d0d527b0fc506d8bfc4ab7b4992c9e027ff0e30737338"} Oct 29 23:10:59.990142 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:10:59.989749 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" event={"ID":"870e234a-d4fd-4877-a458-d8bbfd13b010","Type":"ContainerStarted","Data":"6a9b5560eed66654cfec9e7c4f846c745b2baba1acb459a857b4c7b3cb634d98"} Oct 29 23:11:00.067017 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:00.066957 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" podStartSLOduration=2.066939444 podStartE2EDuration="2.066939444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:00.050915448 +0000 UTC m=+1713.991463358" watchObservedRunningTime="2026-10-29 23:11:00.066939444 +0000 UTC m=+1714.007487353" Oct 29 23:11:00.690971 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:00.690937 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:11:00.692580 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:00.692554 4821 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 29 23:11:00.692728 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:00.692604 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" podUID="870e234a-d4fd-4877-a458-d8bbfd13b010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 29 23:11:01.691095 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:01.691061 4821 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 29 23:11:01.691646 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:01.691116 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" podUID="870e234a-d4fd-4877-a458-d8bbfd13b010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 29 23:11:02.744224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:02.742973 4821 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 29 23:11:02.744224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:02.743073 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" podUID="870e234a-d4fd-4877-a458-d8bbfd13b010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 29 23:11:03.513344 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:03.513297 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:03.703318 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:03.702785 4821 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 29 23:11:03.703318 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:03.702881 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" podUID="870e234a-d4fd-4877-a458-d8bbfd13b010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 29 23:11:03.729286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:03.728618 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:03.763227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:03.761296 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 23:11:03.792287 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:03.767697 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:03.792287 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:03.771293 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:03.792287 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:03.772342 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:03.843267 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:03.835289 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded" Oct 29 23:11:03.947806 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:03.947765 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:03.984371 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:03.984320 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 29 23:11:03.985302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:03.985240 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:04.013156 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:04.008284 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 29 23:11:04.013156 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:04.012742 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 29 23:11:04.017225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:04.015987 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:04.438808 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:04.438764 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:04.451840 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:04.451753 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:04.503317 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:04.503259 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:04.515801 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:04.513733 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:04.515801 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:04.514136 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 23:11:04.560373 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:04.560331 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:04.817287 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:04.798276 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 29 23:11:04.817287 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:04.799907 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 29 23:11:04.843912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:04.821816 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 23:11:04.843912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:04.821893 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 23:11:04.843912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:04.833007 4821 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 29 23:11:04.843912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:04.833057 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" podUID="870e234a-d4fd-4877-a458-d8bbfd13b010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 29 23:11:04.843912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:04.833725 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Oct 29 23:11:04.843912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:04.842847 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 29 23:11:04.843912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:04.843190 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 23:11:04.953396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:04.951683 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:04.953396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:04.952093 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded" Oct 29 23:11:04.953396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:04.952443 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 23:11:04.989336 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:04.989294 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:04.990006 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:04.989612 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:05.040408 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:04.990249 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 23:11:05.040753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:05.040730 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 23:11:05.067658 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:05.026726 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 29 23:11:05.114235 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:05.102630 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 23:11:05.693191 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:05.692438 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:05.748108 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:05.747424 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 23:11:05.834330 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:05.828955 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 29 23:11:05.892700 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:05.892641 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:05.988294 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:05.983620 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 29 23:11:05.988294 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:05.987512 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 23:11:05.988294 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:05.987616 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 23:11:06.026930 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:05.990009 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 29 23:11:06.026930 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:05.990591 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 23:11:06.026930 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:06.000489 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:06.026930 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:06.025490 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:06.039324 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:06.027732 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 29 23:11:06.048227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:06.040543 4821 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 29 23:11:06.048560 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:06.048523 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" podUID="870e234a-d4fd-4877-a458-d8bbfd13b010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 29 23:11:06.082001 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:06.081951 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:06.714279 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:06.714193 4821 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 29 23:11:06.714584 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:06.714555 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" podUID="870e234a-d4fd-4877-a458-d8bbfd13b010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 29 23:11:06.718809 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:06.718466 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:06.730133 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:06.728177 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:06.735979 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:06.735303 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:06.769374 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:06.767650 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:06.777598 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:06.776330 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 23:11:06.821350 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:06.821308 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:06.904227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:06.881580 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:06.959891 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:06.959844 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:07.018853 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:06.990167 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:07.019329 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:07.019274 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:07.047971 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:06.990239 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:07.047971 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:06.990703 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:07.047971 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:07.023721 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 23:11:07.047971 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:07.025718 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:07.047971 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:07.028266 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:07.047971 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:07.028511 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 23:11:07.135323 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:07.064315 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa7f0ef4-s9tsc"] Oct 29 23:11:07.135323 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:07.095990 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:07.135323 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:07.114272 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 29 23:11:07.496367 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:07.496334 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa7f0ef4-s9tsc"] Oct 29 23:11:07.688071 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:07.688036 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 23:11:07.694319 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:07.694281 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:07.738228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:07.721738 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:07.739078 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:07.739041 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:07.739706 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:07.739685 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 23:11:07.741499 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:07.740455 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:07.741997 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:07.741980 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 23:11:07.830409 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:07.830306 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:07.918225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:07.892746 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 23:11:07.918225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:07.885346 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:07.994232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:07.991834 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:07.994232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:07.992337 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:11:07.994232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:07.993526 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:11:08.022268 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:07.995115 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:11:08.051227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:08.050332 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 29 23:11:08.051227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:08.050742 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 29 23:11:08.051227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:08.051007 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 29 23:11:08.051579 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:08.051274 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 29 23:11:08.067248 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:08.055293 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:08.067248 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:08.055676 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 23:11:08.140253 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:08.124451 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:08.597815 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:08.586993 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7a10320-b197-4471-ac63-fd5041d2a8d4" path="/var/lib/kubelet/pods/f7a10320-b197-4471-ac63-fd5041d2a8d4/volumes" Oct 29 23:11:08.597815 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:08.593898 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:08.597815 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:08.597580 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 23:11:08.657227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:08.654748 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 23:11:08.690342 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:08.687863 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:08.734635 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:08.734602 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 23:11:08.740408 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:08.740359 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:08.748528 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:08.748494 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 29 23:11:08.874864 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:08.874834 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 23:11:09.020225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:08.999378 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:09.053360 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:09.053309 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:09.088227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:09.068267 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 29 23:11:09.118676 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:09.088487 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 29 23:11:09.118676 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:09.088789 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 29 23:11:09.176653 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:09.141602 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:09.196664 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:09.195416 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 23:11:09.302776 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:09.302610 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:09.659670 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:09.659613 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 29 23:11:09.757726 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:09.728720 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 29 23:11:09.761120 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:09.752526 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 29 23:11:09.761692 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:09.761665 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 23:11:09.782386 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:09.767639 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:09.789375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:09.789338 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:09.794510 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:09.794485 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 23:11:10.013279 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:10.008005 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:10.013279 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:10.008621 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Oct 29 23:11:10.059362 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:10.059315 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 23:11:10.090343 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:10.090295 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:10.115721 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:10.115678 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:10.274284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:10.096543 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 29 23:11:10.274284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:10.095342 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:10.274284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:10.142388 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:10.274284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:10.224376 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded" Oct 29 23:11:10.309114 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:10.309057 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:10.605976 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:10.605891 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 23:11:10.611679 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:10.611654 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 23:11:10.674087 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:10.673540 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 23:11:10.679299 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:10.679278 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 23:11:10.707395 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:10.706708 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 23:11:10.757675 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:10.757646 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 23:11:10.765226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:10.761882 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 23:11:10.765226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:10.762318 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 29 23:11:10.765226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:10.762768 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 23:11:10.821374 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:10.813051 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 23:11:10.888416 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:10.860343 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:10.888416 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:10.887465 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 23:11:10.924862 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:10.924829 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 23:11:11.050815 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:11.050777 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:11.097146 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:11.097101 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 23:11:12.254790 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:12.254376 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:11:12.254790 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:12.254446 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" podUID="c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:16.035148 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:16.035112 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-850a6259-nkqdq"] Oct 29 23:11:16.046421 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:16.046391 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-850a6259-nkqdq"] Oct 29 23:11:16.178177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:16.178140 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c1e06ef-fc16-4678-a434-b28506de2eb3" path="/var/lib/kubelet/pods/8c1e06ef-fc16-4678-a434-b28506de2eb3/volumes" Oct 29 23:11:17.032938 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:17.032904 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc1eb15-s8ttb"] Oct 29 23:11:17.043764 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:17.043722 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dc1eb15-s8ttb"] Oct 29 23:11:18.256348 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:18.177953 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d33a5db-ede9-4b1c-bc8d-d436e87add8a" path="/var/lib/kubelet/pods/7d33a5db-ede9-4b1c-bc8d-d436e87add8a/volumes" Oct 29 23:11:19.098975 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:19.098943 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": read tcp 10.133.0.2:36704->10.133.0.195:15021: read: connection reset by peer" start-of-body= Oct 29 23:11:19.099279 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:19.099249 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" podUID="b74847df-b26e-4c6b-8f86-b86b5282c004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": read tcp 10.133.0.2:36704->10.133.0.195:15021: read: connection reset by peer" Oct 29 23:11:20.158503 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:20.158469 4821 generic.go:356] "Generic (PLEG): container finished" podID="b74847df-b26e-4c6b-8f86-b86b5282c004" containerID="3f0561ec234de9dda8290ebc9313b4559c76f24e3d91a44c7b6ba8ae61f87ae0" exitCode=0 Oct 29 23:11:20.181322 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:20.158549 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" event={"ID":"b74847df-b26e-4c6b-8f86-b86b5282c004","Type":"ContainerDied","Data":"3f0561ec234de9dda8290ebc9313b4559c76f24e3d91a44c7b6ba8ae61f87ae0"} Oct 29 23:11:20.181322 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:20.159085 4821 scope.go:122] "RemoveContainer" containerID="3f0561ec234de9dda8290ebc9313b4559c76f24e3d91a44c7b6ba8ae61f87ae0" Oct 29 23:11:20.181322 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:20.160602 4821 generic.go:356] "Generic (PLEG): container finished" podID="c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" containerID="d1217ac883fb396639158d9a394a35279bb12c6ba71cb0290eafd90e822f2101" exitCode=0 Oct 29 23:11:20.181322 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:20.160644 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" event={"ID":"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a","Type":"ContainerDied","Data":"d1217ac883fb396639158d9a394a35279bb12c6ba71cb0290eafd90e822f2101"} Oct 29 23:11:20.181322 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:20.161043 4821 scope.go:122] "RemoveContainer" containerID="d1217ac883fb396639158d9a394a35279bb12c6ba71cb0290eafd90e822f2101" Oct 29 23:11:20.181322 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:20.161125 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:11:20.181322 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:20.161220 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:11:20.181322 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:20.161266 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:11:20.181322 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:20.162816 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:11:20.181322 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:20.162877 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:11:20.181322 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:20.162908 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:11:21.176829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:21.176783 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" event={"ID":"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a","Type":"ContainerStarted","Data":"7068df05ddf4843625ed45bbecfe2e575c0ccacc92d93fec4bea2af66e2d2276"} Oct 29 23:11:21.179156 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:21.179124 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" event={"ID":"b74847df-b26e-4c6b-8f86-b86b5282c004","Type":"ContainerStarted","Data":"5d21db15320c85c53f4b47f02ff6286548fd99cdd79c37ee979ec6cf97a18f77"} Oct 29 23:11:21.181793 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:21.181768 4821 generic.go:356] "Generic (PLEG): container finished" podID="f3f2d74f-e211-41ce-8ed6-432892f64365" containerID="646da0945b2d2ef89e14a88a15f2757d0928bb9e4ad2381bf453824fa47e4184" exitCode=0 Oct 29 23:11:21.181891 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:21.181818 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" event={"ID":"f3f2d74f-e211-41ce-8ed6-432892f64365","Type":"ContainerDied","Data":"646da0945b2d2ef89e14a88a15f2757d0928bb9e4ad2381bf453824fa47e4184"} Oct 29 23:11:21.182115 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:21.182102 4821 scope.go:122] "RemoveContainer" containerID="646da0945b2d2ef89e14a88a15f2757d0928bb9e4ad2381bf453824fa47e4184" Oct 29 23:11:21.183706 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:21.183673 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:11:21.183818 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:21.183759 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:11:21.183818 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:21.183809 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:11:21.904485 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:21.904454 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:11:21.905092 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:21.905070 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 29 23:11:21.905184 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:21.905131 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" podUID="c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 29 23:11:21.913350 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:21.913325 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:11:21.913520 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:21.913496 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 29 23:11:21.913697 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:21.913673 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" podUID="b74847df-b26e-4c6b-8f86-b86b5282c004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 29 23:11:22.210094 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:22.210012 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" event={"ID":"f3f2d74f-e211-41ce-8ed6-432892f64365","Type":"ContainerStarted","Data":"3b9f57fe212d1a4ddc05770d5f93e3870c8767479b97db698371e6cd9413b369"} Oct 29 23:11:22.914650 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:22.914617 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 29 23:11:22.914971 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:22.914930 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" podUID="b74847df-b26e-4c6b-8f86-b86b5282c004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 29 23:11:22.943250 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:22.940326 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 29 23:11:22.943250 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:22.940384 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" podUID="c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 29 23:11:23.143661 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:23.143623 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:11:23.157162 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:23.157125 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 29 23:11:23.157498 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:23.157465 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" podUID="f3f2d74f-e211-41ce-8ed6-432892f64365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 29 23:11:23.628236 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:23.627383 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:23.809497 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:23.806340 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 29 23:11:23.828913 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:23.828867 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:23.895286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:23.886302 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:23.913145 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:23.913107 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 29 23:11:23.913459 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:23.913430 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" podUID="c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 29 23:11:23.923229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:23.914185 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 29 23:11:23.923499 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:23.923465 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" podUID="b74847df-b26e-4c6b-8f86-b86b5282c004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 29 23:11:23.929546 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:23.929509 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:24.004170 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:24.003381 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:24.004170 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:24.003820 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 29 23:11:24.023389 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:24.023345 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:24.049985 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:24.049310 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:24.090515 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:11:24.090425 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:11:24.105016 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:24.104015 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 29 23:11:24.138137 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:24.137695 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 29 23:11:24.138137 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:24.137757 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" podUID="f3f2d74f-e211-41ce-8ed6-432892f64365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 29 23:11:24.439962 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:24.439912 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 29 23:11:24.441623 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:24.441491 4821 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded" Oct 29 23:11:24.807803 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:24.807692 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 29 23:11:24.809562 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:24.809541 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 23:11:24.909256 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:24.908726 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 29 23:11:24.909256 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:24.908803 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" podUID="c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 29 23:11:24.913717 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:24.913601 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 29 23:11:24.913717 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:24.913653 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" podUID="b74847df-b26e-4c6b-8f86-b86b5282c004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 29 23:11:25.057357 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:25.057304 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:25.139662 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:25.139619 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 29 23:11:25.139989 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:25.139957 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" podUID="f3f2d74f-e211-41ce-8ed6-432892f64365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 29 23:11:25.205348 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:25.204364 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:11:25.813372 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:25.809326 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 29 23:11:25.911473 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:25.904606 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 29 23:11:25.911473 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:25.904668 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" podUID="c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 29 23:11:25.913599 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:25.913567 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 29 23:11:25.913818 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:25.913791 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" podUID="b74847df-b26e-4c6b-8f86-b86b5282c004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 29 23:11:26.256227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:26.253681 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 29 23:11:26.256227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:26.253745 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" podUID="f3f2d74f-e211-41ce-8ed6-432892f64365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 29 23:11:26.510614 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:26.510532 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:11:26.868465 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:26.868420 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:26.916705 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:26.916668 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 29 23:11:26.917004 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:26.916973 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" podUID="b74847df-b26e-4c6b-8f86-b86b5282c004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 29 23:11:26.919151 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:26.912185 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 29 23:11:26.919340 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:26.919316 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" podUID="c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 29 23:11:26.927518 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:26.927486 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:26.952887 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:26.952857 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 23:11:27.008971 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:27.008928 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:27.014380 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:27.014348 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:27.225283 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:27.199840 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 29 23:11:27.225639 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:27.225592 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" podUID="f3f2d74f-e211-41ce-8ed6-432892f64365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 29 23:11:27.904180 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:27.904147 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 29 23:11:27.904651 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:27.904263 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" podUID="c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 29 23:11:27.913771 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:27.913733 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Oct 29 23:11:27.913979 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:27.913954 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" podUID="b74847df-b26e-4c6b-8f86-b86b5282c004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Oct 29 23:11:28.176410 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:28.171373 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 29 23:11:28.176410 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:28.171435 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" podUID="f3f2d74f-e211-41ce-8ed6-432892f64365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 29 23:11:28.909022 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:28.908990 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:11:28.910258 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:28.910233 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:11:28.926846 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:28.924528 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:11:28.926846 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:28.925864 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:11:28.928434 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:28.928413 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:11:29.073639 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:29.073513 4821 scope.go:122] "RemoveContainer" containerID="9c91e2c67cc9ee56bf23b73dd765ea7ced96a8a95f6ebeb03c533ac9f6128b82" Oct 29 23:11:29.091932 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:29.091908 4821 scope.go:122] "RemoveContainer" containerID="fb72d757cabcc0f1fa3c196a1ab2670a80028107420cf53e3647a6d230970534" Oct 29 23:11:29.106953 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:29.106928 4821 scope.go:122] "RemoveContainer" containerID="e411dda91e8ed10c78b320184303b79528172aaecb550597a37d5e391d609586" Oct 29 23:11:29.144254 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:29.142612 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 29 23:11:29.144254 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:29.142673 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" podUID="f3f2d74f-e211-41ce-8ed6-432892f64365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 29 23:11:30.197302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:30.197269 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:11:30.204427 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:30.204400 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:11:31.705159 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:31.705122 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv"] Oct 29 23:11:31.705996 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:31.705962 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" podUID="870e234a-d4fd-4877-a458-d8bbfd13b010" containerName="istio-proxy" containerID="cri-o://750e7bd23c5129f13d2d0d527b0fc506d8bfc4ab7b4992c9e027ff0e30737338" gracePeriod=30 Oct 29 23:11:31.723796 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:31.723752 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts"] Oct 29 23:11:31.726801 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:31.725786 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" podUID="09716f15-687a-47d5-960a-19beab856a8b" containerName="istio-proxy" containerID="cri-o://9b2658faa70a9462a4b71645c08200925d12dffb7af52a05224af2663661cba7" gracePeriod=30 Oct 29 23:11:35.889166 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:35.889130 4821 patch_prober.go:29] interesting pod/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:11:35.889594 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:35.889187 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" podUID="09716f15-687a-47d5-960a-19beab856a8b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:11:37.069752 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.068895 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:11:37.085343 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.085320 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:11:37.111372 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.111341 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-token\" (UniqueName: \"kubernetes.io/projected/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-token\") pod \"870e234a-d4fd-4877-a458-d8bbfd13b010\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " Oct 29 23:11:37.111559 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.111427 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-podinfo\") pod \"870e234a-d4fd-4877-a458-d8bbfd13b010\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " Oct 29 23:11:37.111625 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.111566 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-workload-certs\") pod \"870e234a-d4fd-4877-a458-d8bbfd13b010\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " Oct 29 23:11:37.111681 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.111625 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/870e234a-d4fd-4877-a458-d8bbfd13b010-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870e234a-d4fd-4877-a458-d8bbfd13b010-istiod-ca-cert\") pod \"870e234a-d4fd-4877-a458-d8bbfd13b010\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " Oct 29 23:11:37.111739 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.111715 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/870e234a-d4fd-4877-a458-d8bbfd13b010-kube-api-access-k4csm\" (UniqueName: \"kubernetes.io/projected/870e234a-d4fd-4877-a458-d8bbfd13b010-kube-api-access-k4csm\") pod \"870e234a-d4fd-4877-a458-d8bbfd13b010\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " Oct 29 23:11:37.111794 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.111751 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-data\") pod \"870e234a-d4fd-4877-a458-d8bbfd13b010\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " Oct 29 23:11:37.111922 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.111900 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-envoy\") pod \"870e234a-d4fd-4877-a458-d8bbfd13b010\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " Oct 29 23:11:37.111922 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.111938 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-credential-socket\") pod \"870e234a-d4fd-4877-a458-d8bbfd13b010\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " Oct 29 23:11:37.112082 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.112046 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-workload-socket\") pod \"870e234a-d4fd-4877-a458-d8bbfd13b010\" (UID: \"870e234a-d4fd-4877-a458-d8bbfd13b010\") " Oct 29 23:11:37.113259 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.113226 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-workload-socket" pod "870e234a-d4fd-4877-a458-d8bbfd13b010" (UID: "870e234a-d4fd-4877-a458-d8bbfd13b010"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:37.113603 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.113579 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/870e234a-d4fd-4877-a458-d8bbfd13b010-istiod-ca-cert" pod "870e234a-d4fd-4877-a458-d8bbfd13b010" (UID: "870e234a-d4fd-4877-a458-d8bbfd13b010"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:37.113828 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.113806 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-workload-certs" pod "870e234a-d4fd-4877-a458-d8bbfd13b010" (UID: "870e234a-d4fd-4877-a458-d8bbfd13b010"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:37.114139 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.114113 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-data" pod "870e234a-d4fd-4877-a458-d8bbfd13b010" (UID: "870e234a-d4fd-4877-a458-d8bbfd13b010"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:37.114795 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.114769 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-credential-socket" pod "870e234a-d4fd-4877-a458-d8bbfd13b010" (UID: "870e234a-d4fd-4877-a458-d8bbfd13b010"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:37.127218 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.125719 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-podinfo" pod "870e234a-d4fd-4877-a458-d8bbfd13b010" (UID: "870e234a-d4fd-4877-a458-d8bbfd13b010"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:11:37.133595 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.133525 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-token" pod "870e234a-d4fd-4877-a458-d8bbfd13b010" (UID: "870e234a-d4fd-4877-a458-d8bbfd13b010"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:37.140380 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.140352 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/870e234a-d4fd-4877-a458-d8bbfd13b010-kube-api-access-k4csm" pod "870e234a-d4fd-4877-a458-d8bbfd13b010" (UID: "870e234a-d4fd-4877-a458-d8bbfd13b010"). InnerVolumeSpecName "kube-api-access-k4csm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:37.145252 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.143657 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-envoy" pod "870e234a-d4fd-4877-a458-d8bbfd13b010" (UID: "870e234a-d4fd-4877-a458-d8bbfd13b010"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:37.216636 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.216598 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-workload-certs\") pod \"09716f15-687a-47d5-960a-19beab856a8b\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " Oct 29 23:11:37.216836 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.216669 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-workload-socket\") pod \"09716f15-687a-47d5-960a-19beab856a8b\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " Oct 29 23:11:37.216836 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.216814 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-istio-data\") pod \"09716f15-687a-47d5-960a-19beab856a8b\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " Oct 29 23:11:37.216967 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.216947 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09716f15-687a-47d5-960a-19beab856a8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09716f15-687a-47d5-960a-19beab856a8b-istio-podinfo\") pod \"09716f15-687a-47d5-960a-19beab856a8b\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " Oct 29 23:11:37.217026 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.216977 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-credential-socket\") pod \"09716f15-687a-47d5-960a-19beab856a8b\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " Oct 29 23:11:37.217026 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.217015 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09716f15-687a-47d5-960a-19beab856a8b-istio-token\" (UniqueName: \"kubernetes.io/projected/09716f15-687a-47d5-960a-19beab856a8b-istio-token\") pod \"09716f15-687a-47d5-960a-19beab856a8b\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " Oct 29 23:11:37.217139 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.217051 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09716f15-687a-47d5-960a-19beab856a8b-kube-api-access-465xj\" (UniqueName: \"kubernetes.io/projected/09716f15-687a-47d5-960a-19beab856a8b-kube-api-access-465xj\") pod \"09716f15-687a-47d5-960a-19beab856a8b\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " Oct 29 23:11:37.217139 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.217080 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09716f15-687a-47d5-960a-19beab856a8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09716f15-687a-47d5-960a-19beab856a8b-istiod-ca-cert\") pod \"09716f15-687a-47d5-960a-19beab856a8b\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " Oct 29 23:11:37.217311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.217230 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-istio-envoy\") pod \"09716f15-687a-47d5-960a-19beab856a8b\" (UID: \"09716f15-687a-47d5-960a-19beab856a8b\") " Oct 29 23:11:37.218288 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.218214 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.218288 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.218247 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.218668 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.218264 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.219732 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.219716 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870e234a-d4fd-4877-a458-d8bbfd13b010-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.219873 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.219735 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4csm\" (UniqueName: \"kubernetes.io/projected/870e234a-d4fd-4877-a458-d8bbfd13b010-kube-api-access-k4csm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.219873 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.219750 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.219873 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.219765 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.219873 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.219777 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.219873 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.219789 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870e234a-d4fd-4877-a458-d8bbfd13b010-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.219873 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.219044 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09716f15-687a-47d5-960a-19beab856a8b-istiod-ca-cert" pod "09716f15-687a-47d5-960a-19beab856a8b" (UID: "09716f15-687a-47d5-960a-19beab856a8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:37.219873 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.219415 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-workload-socket" pod "09716f15-687a-47d5-960a-19beab856a8b" (UID: "09716f15-687a-47d5-960a-19beab856a8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:37.219873 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.219596 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-istio-data" pod "09716f15-687a-47d5-960a-19beab856a8b" (UID: "09716f15-687a-47d5-960a-19beab856a8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:37.219873 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.219606 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-workload-certs" pod "09716f15-687a-47d5-960a-19beab856a8b" (UID: "09716f15-687a-47d5-960a-19beab856a8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:37.220627 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.219889 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-credential-socket" pod "09716f15-687a-47d5-960a-19beab856a8b" (UID: "09716f15-687a-47d5-960a-19beab856a8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:37.223230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.223160 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09716f15-687a-47d5-960a-19beab856a8b-istio-podinfo" pod "09716f15-687a-47d5-960a-19beab856a8b" (UID: "09716f15-687a-47d5-960a-19beab856a8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:11:37.223521 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.223494 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09716f15-687a-47d5-960a-19beab856a8b-kube-api-access-465xj" pod "09716f15-687a-47d5-960a-19beab856a8b" (UID: "09716f15-687a-47d5-960a-19beab856a8b"). InnerVolumeSpecName "kube-api-access-465xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:37.225249 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.223858 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-istio-envoy" pod "09716f15-687a-47d5-960a-19beab856a8b" (UID: "09716f15-687a-47d5-960a-19beab856a8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:11:37.225931 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.225906 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09716f15-687a-47d5-960a-19beab856a8b-istio-token" pod "09716f15-687a-47d5-960a-19beab856a8b" (UID: "09716f15-687a-47d5-960a-19beab856a8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:37.334363 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.333708 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.334363 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.333745 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.334363 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.333759 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.334363 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.333773 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09716f15-687a-47d5-960a-19beab856a8b-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.334363 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.333786 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.334363 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.333799 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09716f15-687a-47d5-960a-19beab856a8b-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.334363 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.333814 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-465xj\" (UniqueName: \"kubernetes.io/projected/09716f15-687a-47d5-960a-19beab856a8b-kube-api-access-465xj\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.334363 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.333827 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09716f15-687a-47d5-960a-19beab856a8b-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.334363 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.333840 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09716f15-687a-47d5-960a-19beab856a8b-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:11:37.659028 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.641054 4821 generic.go:356] "Generic (PLEG): container finished" podID="870e234a-d4fd-4877-a458-d8bbfd13b010" containerID="750e7bd23c5129f13d2d0d527b0fc506d8bfc4ab7b4992c9e027ff0e30737338" exitCode=0 Oct 29 23:11:37.659028 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.641162 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" event={"ID":"870e234a-d4fd-4877-a458-d8bbfd13b010","Type":"ContainerDied","Data":"750e7bd23c5129f13d2d0d527b0fc506d8bfc4ab7b4992c9e027ff0e30737338"} Oct 29 23:11:37.659028 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.641264 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" event={"ID":"870e234a-d4fd-4877-a458-d8bbfd13b010","Type":"ContainerDied","Data":"6a9b5560eed66654cfec9e7c4f846c745b2baba1acb459a857b4c7b3cb634d98"} Oct 29 23:11:37.659028 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.641294 4821 scope.go:122] "RemoveContainer" containerID="750e7bd23c5129f13d2d0d527b0fc506d8bfc4ab7b4992c9e027ff0e30737338" Oct 29 23:11:37.659028 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.641463 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv" Oct 29 23:11:37.659028 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.654589 4821 generic.go:356] "Generic (PLEG): container finished" podID="09716f15-687a-47d5-960a-19beab856a8b" containerID="9b2658faa70a9462a4b71645c08200925d12dffb7af52a05224af2663661cba7" exitCode=0 Oct 29 23:11:37.659028 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.654753 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" event={"ID":"09716f15-687a-47d5-960a-19beab856a8b","Type":"ContainerDied","Data":"9b2658faa70a9462a4b71645c08200925d12dffb7af52a05224af2663661cba7"} Oct 29 23:11:37.659028 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.654821 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" event={"ID":"09716f15-687a-47d5-960a-19beab856a8b","Type":"ContainerDied","Data":"b36198b910d4a3b1a1314e843f79043a1ff56c42da11fc38f78084fe1b9ac3f6"} Oct 29 23:11:37.659028 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.654954 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts" Oct 29 23:11:37.668730 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.668706 4821 scope.go:122] "RemoveContainer" containerID="750e7bd23c5129f13d2d0d527b0fc506d8bfc4ab7b4992c9e027ff0e30737338" Oct 29 23:11:37.669368 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:11:37.669342 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"750e7bd23c5129f13d2d0d527b0fc506d8bfc4ab7b4992c9e027ff0e30737338\": container with ID starting with 750e7bd23c5129f13d2d0d527b0fc506d8bfc4ab7b4992c9e027ff0e30737338 not found: ID does not exist" containerID="750e7bd23c5129f13d2d0d527b0fc506d8bfc4ab7b4992c9e027ff0e30737338" Oct 29 23:11:37.669546 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.669522 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"750e7bd23c5129f13d2d0d527b0fc506d8bfc4ab7b4992c9e027ff0e30737338"} err="failed to get container status \"750e7bd23c5129f13d2d0d527b0fc506d8bfc4ab7b4992c9e027ff0e30737338\": rpc error: code = NotFound desc = could not find container \"750e7bd23c5129f13d2d0d527b0fc506d8bfc4ab7b4992c9e027ff0e30737338\": container with ID starting with 750e7bd23c5129f13d2d0d527b0fc506d8bfc4ab7b4992c9e027ff0e30737338 not found: ID does not exist" Oct 29 23:11:37.669644 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.669634 4821 scope.go:122] "RemoveContainer" containerID="9b2658faa70a9462a4b71645c08200925d12dffb7af52a05224af2663661cba7" Oct 29 23:11:37.701342 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.701309 4821 scope.go:122] "RemoveContainer" containerID="9b2658faa70a9462a4b71645c08200925d12dffb7af52a05224af2663661cba7" Oct 29 23:11:37.701852 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:11:37.701821 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2658faa70a9462a4b71645c08200925d12dffb7af52a05224af2663661cba7\": container with ID starting with 9b2658faa70a9462a4b71645c08200925d12dffb7af52a05224af2663661cba7 not found: ID does not exist" containerID="9b2658faa70a9462a4b71645c08200925d12dffb7af52a05224af2663661cba7" Oct 29 23:11:37.701962 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.701865 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2658faa70a9462a4b71645c08200925d12dffb7af52a05224af2663661cba7"} err="failed to get container status \"9b2658faa70a9462a4b71645c08200925d12dffb7af52a05224af2663661cba7\": rpc error: code = NotFound desc = could not find container \"9b2658faa70a9462a4b71645c08200925d12dffb7af52a05224af2663661cba7\": container with ID starting with 9b2658faa70a9462a4b71645c08200925d12dffb7af52a05224af2663661cba7 not found: ID does not exist" Oct 29 23:11:37.755395 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.755368 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv"] Oct 29 23:11:37.792829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.792795 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-dhqwv"] Oct 29 23:11:37.903238 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.879274 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts"] Oct 29 23:11:37.907221 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:37.904834 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-a9ced3-openshift-default-7d8898bf57-cskts"] Oct 29 23:11:38.211225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:38.203009 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09716f15-687a-47d5-960a-19beab856a8b" path="/var/lib/kubelet/pods/09716f15-687a-47d5-960a-19beab856a8b/volumes" Oct 29 23:11:38.211225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:38.206865 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="870e234a-d4fd-4877-a458-d8bbfd13b010" path="/var/lib/kubelet/pods/870e234a-d4fd-4877-a458-d8bbfd13b010/volumes" Oct 29 23:11:51.351670 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:51.351626 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b85895da-llgbx"] Oct 29 23:11:51.352559 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:51.352533 4821 cpu_manager.go:414] "containerMap: removing container" podUID="870e234a-d4fd-4877-a458-d8bbfd13b010" containerName="istio-proxy" Oct 29 23:11:51.352719 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:51.352701 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="870e234a-d4fd-4877-a458-d8bbfd13b010" containerName="istio-proxy" Oct 29 23:11:51.352805 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:51.352758 4821 cpu_manager.go:414] "containerMap: removing container" podUID="09716f15-687a-47d5-960a-19beab856a8b" containerName="istio-proxy" Oct 29 23:11:51.352805 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:51.352768 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09716f15-687a-47d5-960a-19beab856a8b" containerName="istio-proxy" Oct 29 23:11:51.352975 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:51.352961 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="870e234a-d4fd-4877-a458-d8bbfd13b010" containerName="istio-proxy" Oct 29 23:11:51.353038 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:51.352989 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="09716f15-687a-47d5-960a-19beab856a8b" containerName="istio-proxy" Oct 29 23:11:51.356723 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:51.356702 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85895da-llgbx" Oct 29 23:11:51.385296 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:51.385264 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b85895da-llgbx"] Oct 29 23:11:51.395377 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:51.391718 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:11:51.395377 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:51.391888 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:11:51.427164 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:51.427128 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b496027f-7bfe-475d-be67-d70d3782952d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b85895da-llgbx\" (UID: \"b496027f-7bfe-475d-be67-d70d3782952d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85895da-llgbx" Oct 29 23:11:51.427694 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:51.427664 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b496027f-7bfe-475d-be67-d70d3782952d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b85895da-llgbx\" (UID: \"b496027f-7bfe-475d-be67-d70d3782952d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85895da-llgbx" Oct 29 23:11:51.427824 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:51.427739 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlr4r\" (UniqueName: \"kubernetes.io/projected/b496027f-7bfe-475d-be67-d70d3782952d-kube-api-access-xlr4r\") pod \"maas-api-revoke-sub-b85895da-llgbx\" (UID: \"b496027f-7bfe-475d-be67-d70d3782952d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85895da-llgbx" Oct 29 23:11:51.543224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:51.537709 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b496027f-7bfe-475d-be67-d70d3782952d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b85895da-llgbx\" (UID: \"b496027f-7bfe-475d-be67-d70d3782952d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85895da-llgbx" Oct 29 23:11:51.543224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:51.537793 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlr4r\" (UniqueName: \"kubernetes.io/projected/b496027f-7bfe-475d-be67-d70d3782952d-kube-api-access-xlr4r\") pod \"maas-api-revoke-sub-b85895da-llgbx\" (UID: \"b496027f-7bfe-475d-be67-d70d3782952d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85895da-llgbx" Oct 29 23:11:51.543224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:51.537964 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b496027f-7bfe-475d-be67-d70d3782952d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b85895da-llgbx\" (UID: \"b496027f-7bfe-475d-be67-d70d3782952d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85895da-llgbx" Oct 29 23:11:51.543224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:51.540338 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b496027f-7bfe-475d-be67-d70d3782952d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b85895da-llgbx\" (UID: \"b496027f-7bfe-475d-be67-d70d3782952d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85895da-llgbx" Oct 29 23:11:51.579224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:51.571187 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b496027f-7bfe-475d-be67-d70d3782952d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b85895da-llgbx\" (UID: \"b496027f-7bfe-475d-be67-d70d3782952d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85895da-llgbx" Oct 29 23:11:51.596227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:51.579945 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlr4r\" (UniqueName: \"kubernetes.io/projected/b496027f-7bfe-475d-be67-d70d3782952d-kube-api-access-xlr4r\") pod \"maas-api-revoke-sub-b85895da-llgbx\" (UID: \"b496027f-7bfe-475d-be67-d70d3782952d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85895da-llgbx" Oct 29 23:11:51.675223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:51.674343 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85895da-llgbx" Oct 29 23:11:52.018823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:52.018791 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b85895da-llgbx"] Oct 29 23:11:52.019952 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:11:52.019927 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb496027f_7bfe_475d_be67_d70d3782952d.slice/crio-1f85694e46bead0b71a58249128f7ffcb941a54c09ee981b6976e74d6d902438 WatchSource:0}: Error finding container 1f85694e46bead0b71a58249128f7ffcb941a54c09ee981b6976e74d6d902438: Status 404 returned error can't find the container with id 1f85694e46bead0b71a58249128f7ffcb941a54c09ee981b6976e74d6d902438 Oct 29 23:11:52.826208 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:52.826088 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85895da-llgbx" event={"ID":"b496027f-7bfe-475d-be67-d70d3782952d","Type":"ContainerStarted","Data":"c933808702e895530f39614d0d39c5921e1a1524d5be2eba7855bd85a087e097"} Oct 29 23:11:52.826208 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:52.826176 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85895da-llgbx" event={"ID":"b496027f-7bfe-475d-be67-d70d3782952d","Type":"ContainerStarted","Data":"1f85694e46bead0b71a58249128f7ffcb941a54c09ee981b6976e74d6d902438"} Oct 29 23:11:52.871527 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:52.871425 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85895da-llgbx" podStartSLOduration=1.871411226 podStartE2EDuration="1.871411226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:52.868005007 +0000 UTC m=+1766.808552918" watchObservedRunningTime="2026-10-29 23:11:52.871411226 +0000 UTC m=+1766.811959135" Oct 29 23:11:53.835690 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:53.835652 4821 generic.go:356] "Generic (PLEG): container finished" podID="b496027f-7bfe-475d-be67-d70d3782952d" containerID="c933808702e895530f39614d0d39c5921e1a1524d5be2eba7855bd85a087e097" exitCode=0 Oct 29 23:11:53.836122 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:53.835744 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85895da-llgbx" event={"ID":"b496027f-7bfe-475d-be67-d70d3782952d","Type":"ContainerDied","Data":"c933808702e895530f39614d0d39c5921e1a1524d5be2eba7855bd85a087e097"} Oct 29 23:11:53.890908 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:11:53.890866 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:11:55.019579 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:55.019557 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85895da-llgbx" Oct 29 23:11:55.107685 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:55.107656 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b496027f-7bfe-475d-be67-d70d3782952d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b496027f-7bfe-475d-be67-d70d3782952d-maas-api-xks-ca\") pod \"b496027f-7bfe-475d-be67-d70d3782952d\" (UID: \"b496027f-7bfe-475d-be67-d70d3782952d\") " Oct 29 23:11:55.107893 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:55.107704 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b496027f-7bfe-475d-be67-d70d3782952d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b496027f-7bfe-475d-be67-d70d3782952d-maas-api-openshift-ca\") pod \"b496027f-7bfe-475d-be67-d70d3782952d\" (UID: \"b496027f-7bfe-475d-be67-d70d3782952d\") " Oct 29 23:11:55.107893 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:55.107857 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b496027f-7bfe-475d-be67-d70d3782952d-kube-api-access-xlr4r\" (UniqueName: \"kubernetes.io/projected/b496027f-7bfe-475d-be67-d70d3782952d-kube-api-access-xlr4r\") pod \"b496027f-7bfe-475d-be67-d70d3782952d\" (UID: \"b496027f-7bfe-475d-be67-d70d3782952d\") " Oct 29 23:11:55.108558 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:55.108508 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b496027f-7bfe-475d-be67-d70d3782952d-maas-api-openshift-ca" pod "b496027f-7bfe-475d-be67-d70d3782952d" (UID: "b496027f-7bfe-475d-be67-d70d3782952d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:11:55.108831 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:55.108803 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b496027f-7bfe-475d-be67-d70d3782952d-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:11:55.110306 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:55.110171 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b496027f-7bfe-475d-be67-d70d3782952d-maas-api-xks-ca" pod "b496027f-7bfe-475d-be67-d70d3782952d" (UID: "b496027f-7bfe-475d-be67-d70d3782952d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:11:55.110935 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:55.110910 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b496027f-7bfe-475d-be67-d70d3782952d-kube-api-access-xlr4r" pod "b496027f-7bfe-475d-be67-d70d3782952d" (UID: "b496027f-7bfe-475d-be67-d70d3782952d"). InnerVolumeSpecName "kube-api-access-xlr4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:11:55.210161 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:55.210128 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b496027f-7bfe-475d-be67-d70d3782952d-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:11:55.210161 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:55.210156 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlr4r\" (UniqueName: \"kubernetes.io/projected/b496027f-7bfe-475d-be67-d70d3782952d-kube-api-access-xlr4r\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:11:55.852458 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:55.852423 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85895da-llgbx" Oct 29 23:11:55.852651 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:55.852451 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85895da-llgbx" event={"ID":"b496027f-7bfe-475d-be67-d70d3782952d","Type":"ContainerDied","Data":"1f85694e46bead0b71a58249128f7ffcb941a54c09ee981b6976e74d6d902438"} Oct 29 23:11:55.852651 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:11:55.852513 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f85694e46bead0b71a58249128f7ffcb941a54c09ee981b6976e74d6d902438" Oct 29 23:12:03.161852 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.161814 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr"] Oct 29 23:12:03.162933 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.162902 4821 cpu_manager.go:414] "containerMap: removing container" podUID="b496027f-7bfe-475d-be67-d70d3782952d" containerName="revoke-keys" Oct 29 23:12:03.162933 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.162932 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b496027f-7bfe-475d-be67-d70d3782952d" containerName="revoke-keys" Oct 29 23:12:03.163139 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.163111 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="b496027f-7bfe-475d-be67-d70d3782952d" containerName="revoke-keys" Oct 29 23:12:03.167762 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.167735 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.195903 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.195866 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr"] Oct 29 23:12:03.200534 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.200494 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-c880c7-openshift-default-dockercfg-kzbxs\"" Oct 29 23:12:03.347230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.343304 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-envoy\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.347230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.343416 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istiod-ca-cert\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.347230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.343554 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-credential-socket\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.347230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.343584 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9hxk\" (UniqueName: \"kubernetes.io/projected/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-kube-api-access-t9hxk\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.347230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.343612 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-workload-socket\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.347230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.343679 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-workload-certs\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.347230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.343709 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-token\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.347230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.343776 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-data\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.347230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.343828 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-podinfo\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.448275 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.444963 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-envoy\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.448275 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.445080 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istiod-ca-cert\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.448275 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.445259 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-credential-socket\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.448275 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.445297 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9hxk\" (UniqueName: \"kubernetes.io/projected/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-kube-api-access-t9hxk\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.448275 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.445327 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-workload-socket\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.448275 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.445399 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-workload-certs\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.448275 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.445432 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-token\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.448275 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.445498 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-data\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.448275 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.445553 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-podinfo\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.448275 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.447025 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-workload-certs\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.448275 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.447302 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-workload-socket\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.448275 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.447566 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-data\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.449060 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.448631 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istiod-ca-cert\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.449060 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.448875 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-credential-socket\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.474269 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.472349 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-podinfo\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.486103 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.486065 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-token\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.498881 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.498814 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-envoy\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.519089 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.519057 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9hxk\" (UniqueName: \"kubernetes.io/projected/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-kube-api-access-t9hxk\") pod \"e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:03.795950 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:03.795841 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:04.813564 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:04.813530 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr"] Oct 29 23:12:04.856609 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:04.852352 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:12:04.856609 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:04.852454 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:12:04.856609 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:04.852500 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:12:04.981065 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:04.980817 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" event={"ID":"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b","Type":"ContainerStarted","Data":"8866065111d36643624815c5577f75479e81b7eda2e19f53c2d464788c04aef0"} Oct 29 23:12:06.549016 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:06.548972 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" event={"ID":"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b","Type":"ContainerStarted","Data":"1af6baa0ceabce775e19465651a91c140ce7af5bae38245349547331269625f6"} Oct 29 23:12:06.614914 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:06.614837 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" podStartSLOduration=3.6148183720000002 podStartE2EDuration="3.614818372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:06.61331976 +0000 UTC m=+1780.553867701" watchObservedRunningTime="2026-10-29 23:12:06.614818372 +0000 UTC m=+1780.555366283" Oct 29 23:12:06.796608 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:06.796575 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:06.812999 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:06.812914 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:07.532570 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:07.532530 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:07.533492 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:07.533468 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:12:12.032750 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:12.032712 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a57b94a-rpkhn"] Oct 29 23:12:12.045589 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:12.045565 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a57b94a-rpkhn"] Oct 29 23:12:12.172366 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:12.172334 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eebc6c7-70cb-4417-8db8-d6e0530b7503" path="/var/lib/kubelet/pods/5eebc6c7-70cb-4417-8db8-d6e0530b7503/volumes" Oct 29 23:12:15.842297 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:15.842262 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7"] Oct 29 23:12:15.847584 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:15.847561 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:12:15.876952 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:15.876889 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-b0c3c3\"/\"kube-root-ca.crt\"" Oct 29 23:12:15.876952 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:15.876914 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-b0c3c3\"/\"openshift-service-ca.crt\"" Oct 29 23:12:15.878854 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:15.878818 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-b0c3c3\"/\"auth-test-model-b0c3c3-kserve-self-signed-certs\"" Oct 29 23:12:15.878998 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:15.878878 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-b0c3c3\"/\"default-dockercfg-ksrdt\"" Oct 29 23:12:15.879553 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:15.879528 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7"] Oct 29 23:12:15.884802 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:15.884778 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-tmp-dir\") pod \"auth-test-model-b0c3c3-kserve-8677db688f-w5nk7\" (UID: \"5ef833d5-5335-4391-b2cf-2dd970f88eed\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:12:15.884937 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:15.884896 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-home\") pod \"auth-test-model-b0c3c3-kserve-8677db688f-w5nk7\" (UID: \"5ef833d5-5335-4391-b2cf-2dd970f88eed\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:12:15.884986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:15.884945 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-dshm\") pod \"auth-test-model-b0c3c3-kserve-8677db688f-w5nk7\" (UID: \"5ef833d5-5335-4391-b2cf-2dd970f88eed\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:12:15.885046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:15.884982 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59v4v\" (UniqueName: \"kubernetes.io/projected/5ef833d5-5335-4391-b2cf-2dd970f88eed-kube-api-access-59v4v\") pod \"auth-test-model-b0c3c3-kserve-8677db688f-w5nk7\" (UID: \"5ef833d5-5335-4391-b2cf-2dd970f88eed\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:12:15.885260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:15.885182 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-model-cache\") pod \"auth-test-model-b0c3c3-kserve-8677db688f-w5nk7\" (UID: \"5ef833d5-5335-4391-b2cf-2dd970f88eed\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:12:15.885325 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:15.885257 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ef833d5-5335-4391-b2cf-2dd970f88eed-tls-certs\") pod \"auth-test-model-b0c3c3-kserve-8677db688f-w5nk7\" (UID: \"5ef833d5-5335-4391-b2cf-2dd970f88eed\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:12:15.987526 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:15.987488 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-tmp-dir\") pod \"auth-test-model-b0c3c3-kserve-8677db688f-w5nk7\" (UID: \"5ef833d5-5335-4391-b2cf-2dd970f88eed\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:12:15.987723 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:15.987586 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-home\") pod \"auth-test-model-b0c3c3-kserve-8677db688f-w5nk7\" (UID: \"5ef833d5-5335-4391-b2cf-2dd970f88eed\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:12:15.987723 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:15.987638 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-dshm\") pod \"auth-test-model-b0c3c3-kserve-8677db688f-w5nk7\" (UID: \"5ef833d5-5335-4391-b2cf-2dd970f88eed\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:12:15.987723 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:15.987682 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59v4v\" (UniqueName: \"kubernetes.io/projected/5ef833d5-5335-4391-b2cf-2dd970f88eed-kube-api-access-59v4v\") pod \"auth-test-model-b0c3c3-kserve-8677db688f-w5nk7\" (UID: \"5ef833d5-5335-4391-b2cf-2dd970f88eed\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:12:15.987888 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:15.987779 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-model-cache\") pod \"auth-test-model-b0c3c3-kserve-8677db688f-w5nk7\" (UID: \"5ef833d5-5335-4391-b2cf-2dd970f88eed\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:12:15.987888 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:15.987823 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ef833d5-5335-4391-b2cf-2dd970f88eed-tls-certs\") pod \"auth-test-model-b0c3c3-kserve-8677db688f-w5nk7\" (UID: \"5ef833d5-5335-4391-b2cf-2dd970f88eed\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:12:15.987994 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:15.987918 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-tmp-dir\") pod \"auth-test-model-b0c3c3-kserve-8677db688f-w5nk7\" (UID: \"5ef833d5-5335-4391-b2cf-2dd970f88eed\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:12:15.988051 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:15.988029 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-home\") pod \"auth-test-model-b0c3c3-kserve-8677db688f-w5nk7\" (UID: \"5ef833d5-5335-4391-b2cf-2dd970f88eed\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:12:15.988296 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:15.988279 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-model-cache\") pod \"auth-test-model-b0c3c3-kserve-8677db688f-w5nk7\" (UID: \"5ef833d5-5335-4391-b2cf-2dd970f88eed\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:12:15.991400 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:15.991376 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-dshm\") pod \"auth-test-model-b0c3c3-kserve-8677db688f-w5nk7\" (UID: \"5ef833d5-5335-4391-b2cf-2dd970f88eed\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:12:15.991939 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:15.991907 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ef833d5-5335-4391-b2cf-2dd970f88eed-tls-certs\") pod \"auth-test-model-b0c3c3-kserve-8677db688f-w5nk7\" (UID: \"5ef833d5-5335-4391-b2cf-2dd970f88eed\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:12:16.012819 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:16.012794 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59v4v\" (UniqueName: \"kubernetes.io/projected/5ef833d5-5335-4391-b2cf-2dd970f88eed-kube-api-access-59v4v\") pod \"auth-test-model-b0c3c3-kserve-8677db688f-w5nk7\" (UID: \"5ef833d5-5335-4391-b2cf-2dd970f88eed\") " pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:12:16.201227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:16.200645 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:12:16.632917 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:12:16.632888 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ef833d5_5335_4391_b2cf_2dd970f88eed.slice/crio-c88ef88d37b7a3eef02204829109cbcb783fad35fad6c5c8f97583c6283b00d1 WatchSource:0}: Error finding container c88ef88d37b7a3eef02204829109cbcb783fad35fad6c5c8f97583c6283b00d1: Status 404 returned error can't find the container with id c88ef88d37b7a3eef02204829109cbcb783fad35fad6c5c8f97583c6283b00d1 Oct 29 23:12:16.633048 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:16.633022 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7"] Oct 29 23:12:17.623373 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:17.623338 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" event={"ID":"5ef833d5-5335-4391-b2cf-2dd970f88eed","Type":"ContainerStarted","Data":"691bac9ba5048c3512f1c56c23595eb8318afe7f83e7788782ce9ae4d4ef23b6"} Oct 29 23:12:17.623773 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:17.623386 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" event={"ID":"5ef833d5-5335-4391-b2cf-2dd970f88eed","Type":"ContainerStarted","Data":"c88ef88d37b7a3eef02204829109cbcb783fad35fad6c5c8f97583c6283b00d1"} Oct 29 23:12:17.671385 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:17.671330 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" podStartSLOduration=2.671317149 podStartE2EDuration="2.671317149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:17.666723469 +0000 UTC m=+1791.607271380" watchObservedRunningTime="2026-10-29 23:12:17.671317149 +0000 UTC m=+1791.611865059" Oct 29 23:12:18.202529 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:18.202491 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:12:23.896639 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:12:23.896597 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:12:26.045464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:26.045429 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30791c5-jjf5n"] Oct 29 23:12:26.059834 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:26.059799 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b30791c5-jjf5n"] Oct 29 23:12:26.172158 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:26.172127 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fb36af1-9a9c-45fc-9996-ffa8bfd7d690" path="/var/lib/kubelet/pods/1fb36af1-9a9c-45fc-9996-ffa8bfd7d690/volumes" Oct 29 23:12:26.202507 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:26.202483 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:12:26.215919 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:26.215892 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:12:26.716729 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:26.716700 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:12:29.184078 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:29.184047 4821 scope.go:122] "RemoveContainer" containerID="26333c942d94b47d21471b45f2098f920d342ad1d0ed9ce41e90dc0896eea511" Oct 29 23:12:29.195159 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:29.195139 4821 scope.go:122] "RemoveContainer" containerID="cb4ff358af0460faa4bc34c4180bfba400997e621580404b95a31eaba5b6e002" Oct 29 23:12:29.205930 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:29.205904 4821 scope.go:122] "RemoveContainer" containerID="6913c8e965dfc17346b6f783985fae0facbc5e53e6f590a235c1185b62ae24a8" Oct 29 23:12:29.215819 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:29.215798 4821 scope.go:122] "RemoveContainer" containerID="344e8353e6748bc1d43ea4c3554307d6c38d222ea96c756027bfb940c6b432b1" Oct 29 23:12:29.226042 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:29.226010 4821 scope.go:122] "RemoveContainer" containerID="b60e84b15826677e11a1f4e07365074ff7cca3fd4c8be1242585cb0c1d82d065" Oct 29 23:12:29.235633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:29.235614 4821 scope.go:122] "RemoveContainer" containerID="f099fba34c63e9a0c5c2973192d12abd60fb3e9116869e25153fe9cca1897db2" Oct 29 23:12:31.816072 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:31.816034 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2"] Oct 29 23:12:31.822508 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:31.822484 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:12:31.843982 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:31.843945 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2"] Oct 29 23:12:31.851945 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:31.851913 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-65e2d3\"/\"auth-test-model-65e2d3-kserve-self-signed-certs\"" Oct 29 23:12:31.852270 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:31.852232 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-65e2d3\"/\"default-dockercfg-m687x\"" Oct 29 23:12:31.852270 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:31.852240 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-65e2d3\"/\"openshift-service-ca.crt\"" Oct 29 23:12:31.852480 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:31.852274 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-65e2d3\"/\"kube-root-ca.crt\"" Oct 29 23:12:31.962223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:31.961614 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-home\") pod \"auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2\" (UID: \"d338e5a2-2352-492b-b76b-9b8159baa18f\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:12:31.962223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:31.961962 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-dshm\") pod \"auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2\" (UID: \"d338e5a2-2352-492b-b76b-9b8159baa18f\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:12:31.962223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:31.962003 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5pth\" (UniqueName: \"kubernetes.io/projected/d338e5a2-2352-492b-b76b-9b8159baa18f-kube-api-access-x5pth\") pod \"auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2\" (UID: \"d338e5a2-2352-492b-b76b-9b8159baa18f\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:12:31.962223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:31.962045 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-tmp-dir\") pod \"auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2\" (UID: \"d338e5a2-2352-492b-b76b-9b8159baa18f\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:12:31.962223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:31.962135 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-model-cache\") pod \"auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2\" (UID: \"d338e5a2-2352-492b-b76b-9b8159baa18f\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:12:31.962223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:31.962166 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d338e5a2-2352-492b-b76b-9b8159baa18f-tls-certs\") pod \"auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2\" (UID: \"d338e5a2-2352-492b-b76b-9b8159baa18f\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:12:32.080284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.076862 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-model-cache\") pod \"auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2\" (UID: \"d338e5a2-2352-492b-b76b-9b8159baa18f\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:12:32.080284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.077499 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d338e5a2-2352-492b-b76b-9b8159baa18f-tls-certs\") pod \"auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2\" (UID: \"d338e5a2-2352-492b-b76b-9b8159baa18f\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:12:32.080284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.077545 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-home\") pod \"auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2\" (UID: \"d338e5a2-2352-492b-b76b-9b8159baa18f\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:12:32.080284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.077885 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-dshm\") pod \"auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2\" (UID: \"d338e5a2-2352-492b-b76b-9b8159baa18f\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:12:32.080284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.077920 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5pth\" (UniqueName: \"kubernetes.io/projected/d338e5a2-2352-492b-b76b-9b8159baa18f-kube-api-access-x5pth\") pod \"auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2\" (UID: \"d338e5a2-2352-492b-b76b-9b8159baa18f\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:12:32.080284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.077974 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-tmp-dir\") pod \"auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2\" (UID: \"d338e5a2-2352-492b-b76b-9b8159baa18f\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:12:32.085603 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.085571 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-home\") pod \"auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2\" (UID: \"d338e5a2-2352-492b-b76b-9b8159baa18f\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:12:32.089611 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.089238 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-tmp-dir\") pod \"auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2\" (UID: \"d338e5a2-2352-492b-b76b-9b8159baa18f\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:12:32.102288 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.089872 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-model-cache\") pod \"auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2\" (UID: \"d338e5a2-2352-492b-b76b-9b8159baa18f\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:12:32.103096 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.103069 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d338e5a2-2352-492b-b76b-9b8159baa18f-tls-certs\") pod \"auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2\" (UID: \"d338e5a2-2352-492b-b76b-9b8159baa18f\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:12:32.120783 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.120747 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-dshm\") pod \"auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2\" (UID: \"d338e5a2-2352-492b-b76b-9b8159baa18f\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:12:32.166388 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.164874 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5pth\" (UniqueName: \"kubernetes.io/projected/d338e5a2-2352-492b-b76b-9b8159baa18f-kube-api-access-x5pth\") pod \"auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2\" (UID: \"d338e5a2-2352-492b-b76b-9b8159baa18f\") " pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:12:32.180227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.170296 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g"] Oct 29 23:12:32.187314 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.183313 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.299648 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.290114 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdj27\" (UniqueName: \"kubernetes.io/projected/4df31e57-c10a-4d14-b275-67b4983210ac-kube-api-access-wdj27\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.299648 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.290300 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-workload-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.299648 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.290352 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df31e57-c10a-4d14-b275-67b4983210ac-istio-podinfo\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.299648 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.290502 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df31e57-c10a-4d14-b275-67b4983210ac-istio-token\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.299648 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.290573 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-workload-certs\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.299648 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.290632 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-istio-envoy\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.299648 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.290750 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-credential-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.299648 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.290859 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-istio-data\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.299648 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.290993 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df31e57-c10a-4d14-b275-67b4983210ac-istiod-ca-cert\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.300366 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.300324 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g"] Oct 29 23:12:32.392181 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.392141 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df31e57-c10a-4d14-b275-67b4983210ac-istio-token\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.392418 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.392216 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-workload-certs\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.392494 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.392409 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-istio-envoy\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.392554 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.392498 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-credential-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.392615 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.392575 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-istio-data\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.392615 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.392580 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-workload-certs\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.392731 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.392678 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df31e57-c10a-4d14-b275-67b4983210ac-istiod-ca-cert\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.392796 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.392748 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdj27\" (UniqueName: \"kubernetes.io/projected/4df31e57-c10a-4d14-b275-67b4983210ac-kube-api-access-wdj27\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.392858 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.392808 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-workload-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.392858 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.392840 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-istio-data\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.393328 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.392855 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df31e57-c10a-4d14-b275-67b4983210ac-istio-podinfo\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.393328 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.392911 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-credential-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.393785 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.393597 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-workload-socket\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.394491 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.394397 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df31e57-c10a-4d14-b275-67b4983210ac-istiod-ca-cert\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.396011 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.395988 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-istio-envoy\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.396281 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.396193 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df31e57-c10a-4d14-b275-67b4983210ac-istio-podinfo\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.418441 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.418409 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df31e57-c10a-4d14-b275-67b4983210ac-istio-token\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.418566 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.418552 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdj27\" (UniqueName: \"kubernetes.io/projected/4df31e57-c10a-4d14-b275-67b4983210ac-kube-api-access-wdj27\") pod \"e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.438557 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.438466 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:12:32.563125 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.563089 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:32.755880 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:12:32.755847 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd338e5a2_2352_492b_b76b_9b8159baa18f.slice/crio-ff9aaabe3eaffbe5b698c3771ed7ddc0c4858e6622172e7c272cc8c71613e295 WatchSource:0}: Error finding container ff9aaabe3eaffbe5b698c3771ed7ddc0c4858e6622172e7c272cc8c71613e295: Status 404 returned error can't find the container with id ff9aaabe3eaffbe5b698c3771ed7ddc0c4858e6622172e7c272cc8c71613e295 Oct 29 23:12:32.757024 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.756998 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2"] Oct 29 23:12:32.883958 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:12:32.883929 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df31e57_c10a_4d14_b275_67b4983210ac.slice/crio-b5d0b59f2d66ae8c5455ac61b3953247ab9f110afdbbce3f9578dfbe846cf7b2 WatchSource:0}: Error finding container b5d0b59f2d66ae8c5455ac61b3953247ab9f110afdbbce3f9578dfbe846cf7b2: Status 404 returned error can't find the container with id b5d0b59f2d66ae8c5455ac61b3953247ab9f110afdbbce3f9578dfbe846cf7b2 Oct 29 23:12:32.884732 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.884626 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g"] Oct 29 23:12:32.887098 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.887061 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:12:32.887192 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.887131 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:12:32.887192 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:32.887161 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:12:33.776528 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:33.776446 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" event={"ID":"4df31e57-c10a-4d14-b275-67b4983210ac","Type":"ContainerStarted","Data":"9054de3ab5d9fc2b4b84224f9d56ecaed4483ced8ba6003bac968ae4f646bff4"} Oct 29 23:12:33.776528 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:33.776517 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" event={"ID":"4df31e57-c10a-4d14-b275-67b4983210ac","Type":"ContainerStarted","Data":"b5d0b59f2d66ae8c5455ac61b3953247ab9f110afdbbce3f9578dfbe846cf7b2"} Oct 29 23:12:33.779256 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:33.779189 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" event={"ID":"d338e5a2-2352-492b-b76b-9b8159baa18f","Type":"ContainerStarted","Data":"7cd6ff4ee8abeb0d1be0599df27b6bdc5bb82912c8750fe3e9fd2156dcaec48c"} Oct 29 23:12:33.779256 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:33.779245 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" event={"ID":"d338e5a2-2352-492b-b76b-9b8159baa18f","Type":"ContainerStarted","Data":"ff9aaabe3eaffbe5b698c3771ed7ddc0c4858e6622172e7c272cc8c71613e295"} Oct 29 23:12:33.890635 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:33.890580 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" podStartSLOduration=1.890564253 podStartE2EDuration="1.890564253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:33.845326468 +0000 UTC m=+1807.785874379" watchObservedRunningTime="2026-10-29 23:12:33.890564253 +0000 UTC m=+1807.831112203" Oct 29 23:12:33.894129 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:33.894085 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" podStartSLOduration=2.894071149 podStartE2EDuration="2.894071149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:33.890460856 +0000 UTC m=+1807.831008766" watchObservedRunningTime="2026-10-29 23:12:33.894071149 +0000 UTC m=+1807.834619106" Oct 29 23:12:34.439457 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:34.439404 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:12:34.563298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:34.563263 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:34.564979 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:34.564952 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 29 23:12:34.565116 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:34.565002 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" podUID="4df31e57-c10a-4d14-b275-67b4983210ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 29 23:12:35.564503 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:35.564465 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 29 23:12:35.564917 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:35.564527 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" podUID="4df31e57-c10a-4d14-b275-67b4983210ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 29 23:12:36.564367 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:36.564331 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 29 23:12:36.564553 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:36.564386 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" podUID="4df31e57-c10a-4d14-b275-67b4983210ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 29 23:12:37.563788 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:37.563746 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 29 23:12:37.564005 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:37.563808 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" podUID="4df31e57-c10a-4d14-b275-67b4983210ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 29 23:12:38.565985 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:38.565948 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 29 23:12:38.566650 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:38.566617 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" podUID="4df31e57-c10a-4d14-b275-67b4983210ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 29 23:12:39.564106 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:39.564069 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 29 23:12:39.564344 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:39.564129 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" podUID="4df31e57-c10a-4d14-b275-67b4983210ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 29 23:12:40.744684 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:40.736534 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:40.744684 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:40.738169 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:40.761575 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:40.761098 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:12:42.447989 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:42.447952 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:12:42.526527 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:42.526491 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:12:43.424975 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:43.424928 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:12:48.325945 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.324803 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4"] Oct 29 23:12:48.339284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.339254 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:12:48.353480 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.352378 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4"] Oct 29 23:12:48.368749 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.368723 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-67363b\"/\"auth-test-model-67363b-kserve-self-signed-certs\"" Oct 29 23:12:48.369160 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.369140 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-67363b\"/\"default-dockercfg-2wwt5\"" Oct 29 23:12:48.383614 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.383580 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-67363b\"/\"openshift-service-ca.crt\"" Oct 29 23:12:48.383834 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.383810 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-67363b\"/\"kube-root-ca.crt\"" Oct 29 23:12:48.520290 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.520253 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-home\") pod \"auth-test-model-67363b-kserve-5df8986f7-5zdc4\" (UID: \"6a19fff5-f4be-494c-97ad-3520546346d1\") " pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:12:48.520466 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.520327 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a19fff5-f4be-494c-97ad-3520546346d1-tls-certs\") pod \"auth-test-model-67363b-kserve-5df8986f7-5zdc4\" (UID: \"6a19fff5-f4be-494c-97ad-3520546346d1\") " pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:12:48.520579 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.520548 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch8t6\" (UniqueName: \"kubernetes.io/projected/6a19fff5-f4be-494c-97ad-3520546346d1-kube-api-access-ch8t6\") pod \"auth-test-model-67363b-kserve-5df8986f7-5zdc4\" (UID: \"6a19fff5-f4be-494c-97ad-3520546346d1\") " pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:12:48.520718 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.520701 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-tmp-dir\") pod \"auth-test-model-67363b-kserve-5df8986f7-5zdc4\" (UID: \"6a19fff5-f4be-494c-97ad-3520546346d1\") " pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:12:48.520783 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.520742 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-model-cache\") pod \"auth-test-model-67363b-kserve-5df8986f7-5zdc4\" (UID: \"6a19fff5-f4be-494c-97ad-3520546346d1\") " pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:12:48.520783 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.520769 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-dshm\") pod \"auth-test-model-67363b-kserve-5df8986f7-5zdc4\" (UID: \"6a19fff5-f4be-494c-97ad-3520546346d1\") " pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:12:48.625229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.624371 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-home\") pod \"auth-test-model-67363b-kserve-5df8986f7-5zdc4\" (UID: \"6a19fff5-f4be-494c-97ad-3520546346d1\") " pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:12:48.625229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.624460 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a19fff5-f4be-494c-97ad-3520546346d1-tls-certs\") pod \"auth-test-model-67363b-kserve-5df8986f7-5zdc4\" (UID: \"6a19fff5-f4be-494c-97ad-3520546346d1\") " pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:12:48.625229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.624576 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8t6\" (UniqueName: \"kubernetes.io/projected/6a19fff5-f4be-494c-97ad-3520546346d1-kube-api-access-ch8t6\") pod \"auth-test-model-67363b-kserve-5df8986f7-5zdc4\" (UID: \"6a19fff5-f4be-494c-97ad-3520546346d1\") " pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:12:48.625229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.624680 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-tmp-dir\") pod \"auth-test-model-67363b-kserve-5df8986f7-5zdc4\" (UID: \"6a19fff5-f4be-494c-97ad-3520546346d1\") " pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:12:48.625229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.624719 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-model-cache\") pod \"auth-test-model-67363b-kserve-5df8986f7-5zdc4\" (UID: \"6a19fff5-f4be-494c-97ad-3520546346d1\") " pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:12:48.625229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.624746 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-dshm\") pod \"auth-test-model-67363b-kserve-5df8986f7-5zdc4\" (UID: \"6a19fff5-f4be-494c-97ad-3520546346d1\") " pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:12:48.632060 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.625989 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-home\") pod \"auth-test-model-67363b-kserve-5df8986f7-5zdc4\" (UID: \"6a19fff5-f4be-494c-97ad-3520546346d1\") " pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:12:48.632060 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.626492 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-tmp-dir\") pod \"auth-test-model-67363b-kserve-5df8986f7-5zdc4\" (UID: \"6a19fff5-f4be-494c-97ad-3520546346d1\") " pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:12:48.632060 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.626726 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-model-cache\") pod \"auth-test-model-67363b-kserve-5df8986f7-5zdc4\" (UID: \"6a19fff5-f4be-494c-97ad-3520546346d1\") " pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:12:48.651230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.633538 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a19fff5-f4be-494c-97ad-3520546346d1-tls-certs\") pod \"auth-test-model-67363b-kserve-5df8986f7-5zdc4\" (UID: \"6a19fff5-f4be-494c-97ad-3520546346d1\") " pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:12:48.651230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.642742 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-dshm\") pod \"auth-test-model-67363b-kserve-5df8986f7-5zdc4\" (UID: \"6a19fff5-f4be-494c-97ad-3520546346d1\") " pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:12:48.677240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.665309 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch8t6\" (UniqueName: \"kubernetes.io/projected/6a19fff5-f4be-494c-97ad-3520546346d1-kube-api-access-ch8t6\") pod \"auth-test-model-67363b-kserve-5df8986f7-5zdc4\" (UID: \"6a19fff5-f4be-494c-97ad-3520546346d1\") " pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:12:48.677240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:48.673494 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:12:49.236175 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:49.236142 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4"] Oct 29 23:12:49.236341 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:12:49.236269 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a19fff5_f4be_494c_97ad_3520546346d1.slice/crio-2c3ea3f232af994abc4411a50a41ae5488e2c279a5b72b8a92e154419c68d999 WatchSource:0}: Error finding container 2c3ea3f232af994abc4411a50a41ae5488e2c279a5b72b8a92e154419c68d999: Status 404 returned error can't find the container with id 2c3ea3f232af994abc4411a50a41ae5488e2c279a5b72b8a92e154419c68d999 Oct 29 23:12:49.301586 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:49.301519 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" event={"ID":"6a19fff5-f4be-494c-97ad-3520546346d1","Type":"ContainerStarted","Data":"2c3ea3f232af994abc4411a50a41ae5488e2c279a5b72b8a92e154419c68d999"} Oct 29 23:12:50.318632 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:50.318594 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" event={"ID":"6a19fff5-f4be-494c-97ad-3520546346d1","Type":"ContainerStarted","Data":"9cce07ab2eb3fb071565ff3268fa6e8adb951ebee99c3a3ead636a415cf2676d"} Oct 29 23:12:50.372090 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:50.372019 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" podStartSLOduration=2.371995913 podStartE2EDuration="2.371995913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:50.363047823 +0000 UTC m=+1824.303595734" watchObservedRunningTime="2026-10-29 23:12:50.371995913 +0000 UTC m=+1824.312543824" Oct 29 23:12:50.676983 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:50.676931 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:12:53.686820 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.686783 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf"] Oct 29 23:12:53.692282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.692173 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:12:53.731955 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.730528 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-755453\"/\"openshift-service-ca.crt\"" Oct 29 23:12:53.731955 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.730551 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-755453\"/\"default-dockercfg-r5lmt\"" Oct 29 23:12:53.731955 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.730554 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-755453\"/\"kube-root-ca.crt\"" Oct 29 23:12:53.731955 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.730532 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-755453\"/\"auth-test-model-755453-kserve-self-signed-certs\"" Oct 29 23:12:53.737694 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.737667 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf"] Oct 29 23:12:53.769255 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.769219 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-tls-certs\") pod \"auth-test-model-755453-kserve-75c564dd45-98jzf\" (UID: \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\") " pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:12:53.769408 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.769391 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-tmp-dir\") pod \"auth-test-model-755453-kserve-75c564dd45-98jzf\" (UID: \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\") " pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:12:53.769553 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.769527 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttszj\" (UniqueName: \"kubernetes.io/projected/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-kube-api-access-ttszj\") pod \"auth-test-model-755453-kserve-75c564dd45-98jzf\" (UID: \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\") " pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:12:53.769604 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.769592 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-home\") pod \"auth-test-model-755453-kserve-75c564dd45-98jzf\" (UID: \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\") " pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:12:53.769680 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.769658 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-dshm\") pod \"auth-test-model-755453-kserve-75c564dd45-98jzf\" (UID: \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\") " pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:12:53.769803 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.769786 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-model-cache\") pod \"auth-test-model-755453-kserve-75c564dd45-98jzf\" (UID: \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\") " pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:12:53.871838 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.871804 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-tls-certs\") pod \"auth-test-model-755453-kserve-75c564dd45-98jzf\" (UID: \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\") " pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:12:53.872065 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.871948 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-tmp-dir\") pod \"auth-test-model-755453-kserve-75c564dd45-98jzf\" (UID: \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\") " pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:12:53.872065 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.872040 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttszj\" (UniqueName: \"kubernetes.io/projected/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-kube-api-access-ttszj\") pod \"auth-test-model-755453-kserve-75c564dd45-98jzf\" (UID: \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\") " pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:12:53.872192 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.872083 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-home\") pod \"auth-test-model-755453-kserve-75c564dd45-98jzf\" (UID: \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\") " pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:12:53.872192 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.872110 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-dshm\") pod \"auth-test-model-755453-kserve-75c564dd45-98jzf\" (UID: \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\") " pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:12:53.872192 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.872220 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-model-cache\") pod \"auth-test-model-755453-kserve-75c564dd45-98jzf\" (UID: \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\") " pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:12:53.872573 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.872548 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-tmp-dir\") pod \"auth-test-model-755453-kserve-75c564dd45-98jzf\" (UID: \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\") " pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:12:53.872663 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.872618 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-home\") pod \"auth-test-model-755453-kserve-75c564dd45-98jzf\" (UID: \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\") " pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:12:53.872720 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.872676 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-model-cache\") pod \"auth-test-model-755453-kserve-75c564dd45-98jzf\" (UID: \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\") " pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:12:53.877311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.875866 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-tls-certs\") pod \"auth-test-model-755453-kserve-75c564dd45-98jzf\" (UID: \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\") " pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:12:53.877311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.876393 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-dshm\") pod \"auth-test-model-755453-kserve-75c564dd45-98jzf\" (UID: \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\") " pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:12:53.896619 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:12:53.896577 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:12:53.900272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:53.900243 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttszj\" (UniqueName: \"kubernetes.io/projected/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-kube-api-access-ttszj\") pod \"auth-test-model-755453-kserve-75c564dd45-98jzf\" (UID: \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\") " pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:12:54.009787 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:54.009690 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:12:54.566943 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:54.566904 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf"] Oct 29 23:12:54.567163 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:12:54.567140 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2aeb91d0_8e92_4c73_ae62_767cdb2e6f84.slice/crio-38a0b357d1680e2e25e367552289d827c61470446ed4727454ea9593fb1480b3 WatchSource:0}: Error finding container 38a0b357d1680e2e25e367552289d827c61470446ed4727454ea9593fb1480b3: Status 404 returned error can't find the container with id 38a0b357d1680e2e25e367552289d827c61470446ed4727454ea9593fb1480b3 Oct 29 23:12:55.392164 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:55.392129 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" event={"ID":"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84","Type":"ContainerStarted","Data":"8befdf747f801f2cad604269aaf9fa4a11c4509dbe4a4e3ff4f9d40c66616b8a"} Oct 29 23:12:55.392653 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:55.392179 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" event={"ID":"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84","Type":"ContainerStarted","Data":"38a0b357d1680e2e25e367552289d827c61470446ed4727454ea9593fb1480b3"} Oct 29 23:12:55.441857 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:55.441800 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" podStartSLOduration=2.441786367 podStartE2EDuration="2.441786367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:55.438461063 +0000 UTC m=+1829.379008974" watchObservedRunningTime="2026-10-29 23:12:55.441786367 +0000 UTC m=+1829.382334303" Oct 29 23:12:56.009987 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:56.009955 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:12:58.676824 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:58.676789 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:12:58.690411 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:58.690385 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:12:59.442966 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:12:59.442924 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:13:04.010140 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:04.010111 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:13:04.022548 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:04.022527 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:13:04.478541 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:04.478506 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:13:11.862607 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:11.862569 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp"] Oct 29 23:13:11.869985 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:11.869946 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:13:11.892467 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:11.890660 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp"] Oct 29 23:13:11.895407 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:11.895378 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-ed3bf2\"/\"default-dockercfg-g7m9d\"" Oct 29 23:13:11.895588 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:11.895520 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-ed3bf2\"/\"kube-root-ca.crt\"" Oct 29 23:13:11.903931 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:11.903907 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-ed3bf2\"/\"openshift-service-ca.crt\"" Oct 29 23:13:11.904071 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:11.903959 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-ed3bf2\"/\"auth-test-model-ed3bf2-kserve-self-signed-certs\"" Oct 29 23:13:12.023457 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:12.023418 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89e6c6b4-ed1a-4f8a-985f-238f0116574a-tls-certs\") pod \"auth-test-model-ed3bf2-kserve-d9564c669-qbftp\" (UID: \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:13:12.023672 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:12.023606 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-home\") pod \"auth-test-model-ed3bf2-kserve-d9564c669-qbftp\" (UID: \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:13:12.023748 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:12.023713 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-model-cache\") pod \"auth-test-model-ed3bf2-kserve-d9564c669-qbftp\" (UID: \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:13:12.023817 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:12.023800 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-tmp-dir\") pod \"auth-test-model-ed3bf2-kserve-d9564c669-qbftp\" (UID: \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:13:12.024031 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:12.024008 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-dshm\") pod \"auth-test-model-ed3bf2-kserve-d9564c669-qbftp\" (UID: \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:13:12.024194 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:12.024173 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqjhm\" (UniqueName: \"kubernetes.io/projected/89e6c6b4-ed1a-4f8a-985f-238f0116574a-kube-api-access-pqjhm\") pod \"auth-test-model-ed3bf2-kserve-d9564c669-qbftp\" (UID: \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:13:12.127274 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:12.126331 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-dshm\") pod \"auth-test-model-ed3bf2-kserve-d9564c669-qbftp\" (UID: \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:13:12.127274 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:12.126391 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqjhm\" (UniqueName: \"kubernetes.io/projected/89e6c6b4-ed1a-4f8a-985f-238f0116574a-kube-api-access-pqjhm\") pod \"auth-test-model-ed3bf2-kserve-d9564c669-qbftp\" (UID: \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:13:12.127274 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:12.126536 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89e6c6b4-ed1a-4f8a-985f-238f0116574a-tls-certs\") pod \"auth-test-model-ed3bf2-kserve-d9564c669-qbftp\" (UID: \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:13:12.127274 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:12.126705 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-home\") pod \"auth-test-model-ed3bf2-kserve-d9564c669-qbftp\" (UID: \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:13:12.127274 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:12.126808 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-model-cache\") pod \"auth-test-model-ed3bf2-kserve-d9564c669-qbftp\" (UID: \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:13:12.127274 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:12.126893 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-tmp-dir\") pod \"auth-test-model-ed3bf2-kserve-d9564c669-qbftp\" (UID: \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:13:12.127771 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:12.127542 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-tmp-dir\") pod \"auth-test-model-ed3bf2-kserve-d9564c669-qbftp\" (UID: \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:13:12.132219 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:12.128539 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-home\") pod \"auth-test-model-ed3bf2-kserve-d9564c669-qbftp\" (UID: \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:13:12.132219 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:12.131665 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-model-cache\") pod \"auth-test-model-ed3bf2-kserve-d9564c669-qbftp\" (UID: \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:13:12.164882 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:12.164848 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-dshm\") pod \"auth-test-model-ed3bf2-kserve-d9564c669-qbftp\" (UID: \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:13:12.169089 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:12.169063 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89e6c6b4-ed1a-4f8a-985f-238f0116574a-tls-certs\") pod \"auth-test-model-ed3bf2-kserve-d9564c669-qbftp\" (UID: \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:13:12.191232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:12.191140 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqjhm\" (UniqueName: \"kubernetes.io/projected/89e6c6b4-ed1a-4f8a-985f-238f0116574a-kube-api-access-pqjhm\") pod \"auth-test-model-ed3bf2-kserve-d9564c669-qbftp\" (UID: \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\") " pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:13:12.217229 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:12.216521 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:13:12.834058 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:13:12.834026 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89e6c6b4_ed1a_4f8a_985f_238f0116574a.slice/crio-d7072dbc980f054cc7d9b17aa268d0eb243e968419b5b90711b0c8bbeb791894 WatchSource:0}: Error finding container d7072dbc980f054cc7d9b17aa268d0eb243e968419b5b90711b0c8bbeb791894: Status 404 returned error can't find the container with id d7072dbc980f054cc7d9b17aa268d0eb243e968419b5b90711b0c8bbeb791894 Oct 29 23:13:12.835038 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:12.834972 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp"] Oct 29 23:13:13.702255 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:13.696970 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" event={"ID":"89e6c6b4-ed1a-4f8a-985f-238f0116574a","Type":"ContainerStarted","Data":"a6f38b54e994a0d6309eba5537dd87a72339946e0e2d5e109c202e18096e9cb0"} Oct 29 23:13:13.702255 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:13.697028 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" event={"ID":"89e6c6b4-ed1a-4f8a-985f-238f0116574a","Type":"ContainerStarted","Data":"d7072dbc980f054cc7d9b17aa268d0eb243e968419b5b90711b0c8bbeb791894"} Oct 29 23:13:13.746240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:13.744386 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" podStartSLOduration=2.744355103 podStartE2EDuration="2.744355103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:13.743597171 +0000 UTC m=+1847.684145077" watchObservedRunningTime="2026-10-29 23:13:13.744355103 +0000 UTC m=+1847.684903014" Oct 29 23:13:14.219393 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:14.219145 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:13:21.033886 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:21.033852 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4cbda27-vpx9w"] Oct 29 23:13:21.046235 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:21.046191 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4cbda27-vpx9w"] Oct 29 23:13:22.173469 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:22.173432 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52620065-ff2f-4acc-b33d-3e283eb5076e" path="/var/lib/kubelet/pods/52620065-ff2f-4acc-b33d-3e283eb5076e/volumes" Oct 29 23:13:22.219125 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:22.219090 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:13:22.234130 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:22.234101 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:13:22.841981 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:22.841951 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:13:23.573371 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:23.573314 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:13:23.573905 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:23.573427 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" podUID="dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:13:23.936292 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:13:23.936252 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:13:29.305165 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:29.305127 4821 scope.go:122] "RemoveContainer" containerID="b8f3ada00cb1300877dfff594f7b571b8147e10d7232bffbf03f08678ff785df" Oct 29 23:13:29.316861 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:29.316832 4821 scope.go:122] "RemoveContainer" containerID="56dcaf3bbee7514ef148d2832897f5ded7bff1af54e0494dc4ffc89e97efc7a9" Oct 29 23:13:29.330434 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:29.330412 4821 scope.go:122] "RemoveContainer" containerID="c6b8f16f620933cacc22778a070f3b5c8865f405c31857611d2b52033d5092ff" Oct 29 23:13:29.340770 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:29.340734 4821 scope.go:122] "RemoveContainer" containerID="fd1c3bcc5eeab13db4e5d666685e8a41cebcd26389afe4602e85cfd0e3f4a743" Oct 29 23:13:31.019996 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:31.019961 4821 generic.go:356] "Generic (PLEG): container finished" podID="dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" containerID="1af6baa0ceabce775e19465651a91c140ce7af5bae38245349547331269625f6" exitCode=0 Oct 29 23:13:31.020613 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:31.020044 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" event={"ID":"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b","Type":"ContainerDied","Data":"1af6baa0ceabce775e19465651a91c140ce7af5bae38245349547331269625f6"} Oct 29 23:13:31.020694 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:31.020676 4821 scope.go:122] "RemoveContainer" containerID="1af6baa0ceabce775e19465651a91c140ce7af5bae38245349547331269625f6" Oct 29 23:13:31.022668 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:31.022643 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:13:31.022782 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:31.022715 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:13:31.022782 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:31.022753 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:13:32.073195 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:32.073144 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" event={"ID":"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b","Type":"ContainerStarted","Data":"f0547326d2511f281bd162c96e3a31ed32544670f6847e8215d9a871fbe822d2"} Oct 29 23:13:32.454123 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:32.454086 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl"] Oct 29 23:13:32.459490 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:32.459469 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl" Oct 29 23:13:32.473194 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:32.473167 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl"] Oct 29 23:13:32.475775 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:32.475753 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-5e52e0\"" Oct 29 23:13:32.475911 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:32.475890 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-5e52e0-dockercfg-knhk7\"" Oct 29 23:13:32.616231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:32.615422 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04c222af-dd88-48ed-88a0-ad27b981b410-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl\" (UID: \"04c222af-dd88-48ed-88a0-ad27b981b410\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl" Oct 29 23:13:32.616231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:32.615543 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbn66\" (UniqueName: \"kubernetes.io/projected/04c222af-dd88-48ed-88a0-ad27b981b410-kube-api-access-cbn66\") pod \"payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl\" (UID: \"04c222af-dd88-48ed-88a0-ad27b981b410\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl" Oct 29 23:13:32.783338 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:32.783246 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04c222af-dd88-48ed-88a0-ad27b981b410-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl\" (UID: \"04c222af-dd88-48ed-88a0-ad27b981b410\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl" Oct 29 23:13:32.784504 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:32.784377 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04c222af-dd88-48ed-88a0-ad27b981b410-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl\" (UID: \"04c222af-dd88-48ed-88a0-ad27b981b410\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl" Oct 29 23:13:32.785239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:32.785181 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbn66\" (UniqueName: \"kubernetes.io/projected/04c222af-dd88-48ed-88a0-ad27b981b410-kube-api-access-cbn66\") pod \"payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl\" (UID: \"04c222af-dd88-48ed-88a0-ad27b981b410\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl" Oct 29 23:13:32.819732 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:32.798720 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:13:32.838222 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:32.823326 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 29 23:13:32.838222 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:32.823386 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" podUID="dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 29 23:13:32.857736 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:32.857403 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbn66\" (UniqueName: \"kubernetes.io/projected/04c222af-dd88-48ed-88a0-ad27b981b410-kube-api-access-cbn66\") pod \"payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl\" (UID: \"04c222af-dd88-48ed-88a0-ad27b981b410\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl" Oct 29 23:13:32.888234 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:32.885716 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w"] Oct 29 23:13:32.894568 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:32.894538 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w" Oct 29 23:13:32.898865 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:32.898839 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w"] Oct 29 23:13:32.999119 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:32.999077 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153f58c3-7d4f-4aa1-b7a5-2f807a0b1811-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w\" (UID: \"153f58c3-7d4f-4aa1-b7a5-2f807a0b1811\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w" Oct 29 23:13:32.999337 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:32.999240 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5656\" (UniqueName: \"kubernetes.io/projected/153f58c3-7d4f-4aa1-b7a5-2f807a0b1811-kube-api-access-d5656\") pod \"payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w\" (UID: \"153f58c3-7d4f-4aa1-b7a5-2f807a0b1811\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w" Oct 29 23:13:33.080376 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:33.080284 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl" Oct 29 23:13:33.100666 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:33.100634 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153f58c3-7d4f-4aa1-b7a5-2f807a0b1811-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w\" (UID: \"153f58c3-7d4f-4aa1-b7a5-2f807a0b1811\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w" Oct 29 23:13:33.100803 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:33.100705 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5656\" (UniqueName: \"kubernetes.io/projected/153f58c3-7d4f-4aa1-b7a5-2f807a0b1811-kube-api-access-d5656\") pod \"payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w\" (UID: \"153f58c3-7d4f-4aa1-b7a5-2f807a0b1811\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w" Oct 29 23:13:33.101598 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:33.101570 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153f58c3-7d4f-4aa1-b7a5-2f807a0b1811-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w\" (UID: \"153f58c3-7d4f-4aa1-b7a5-2f807a0b1811\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w" Oct 29 23:13:33.143282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:33.143193 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5656\" (UniqueName: \"kubernetes.io/projected/153f58c3-7d4f-4aa1-b7a5-2f807a0b1811-kube-api-access-d5656\") pod \"payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w\" (UID: \"153f58c3-7d4f-4aa1-b7a5-2f807a0b1811\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w" Oct 29 23:13:33.236224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:33.235402 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w" Oct 29 23:13:33.677514 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:33.677484 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl"] Oct 29 23:13:33.677826 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:13:33.677799 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c222af_dd88_48ed_88a0_ad27b981b410.slice/crio-1711df1630232cab6c760c2845d0c5bd15c1dbf386a80a37a60bac39b16a280e WatchSource:0}: Error finding container 1711df1630232cab6c760c2845d0c5bd15c1dbf386a80a37a60bac39b16a280e: Status 404 returned error can't find the container with id 1711df1630232cab6c760c2845d0c5bd15c1dbf386a80a37a60bac39b16a280e Oct 29 23:13:33.691602 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:33.691577 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w"] Oct 29 23:13:33.694491 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:13:33.694465 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153f58c3_7d4f_4aa1_b7a5_2f807a0b1811.slice/crio-b67916c139e5dff29ec6cbfbbdcf884828c8ce677e007066d4511bd3c81eb936 WatchSource:0}: Error finding container b67916c139e5dff29ec6cbfbbdcf884828c8ce677e007066d4511bd3c81eb936: Status 404 returned error can't find the container with id b67916c139e5dff29ec6cbfbbdcf884828c8ce677e007066d4511bd3c81eb936 Oct 29 23:13:33.798117 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:33.798092 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 29 23:13:33.798255 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:33.798141 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" podUID="dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 29 23:13:34.105322 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:34.105231 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w" event={"ID":"153f58c3-7d4f-4aa1-b7a5-2f807a0b1811","Type":"ContainerStarted","Data":"31e63dfd0f32f3353f7aac3bdf9f0d3f9818e8aa741b00920de6ef688bbc72a3"} Oct 29 23:13:34.105322 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:34.105291 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w" event={"ID":"153f58c3-7d4f-4aa1-b7a5-2f807a0b1811","Type":"ContainerStarted","Data":"b67916c139e5dff29ec6cbfbbdcf884828c8ce677e007066d4511bd3c81eb936"} Oct 29 23:13:34.105812 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:34.105360 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w" Oct 29 23:13:34.107507 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:34.107480 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl" event={"ID":"04c222af-dd88-48ed-88a0-ad27b981b410","Type":"ContainerStarted","Data":"35648b56f84d8cd533719f5598611af5aa077dab1987010e5bea4d17e0858ccf"} Oct 29 23:13:34.107639 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:34.107527 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl" event={"ID":"04c222af-dd88-48ed-88a0-ad27b981b410","Type":"ContainerStarted","Data":"1711df1630232cab6c760c2845d0c5bd15c1dbf386a80a37a60bac39b16a280e"} Oct 29 23:13:34.107639 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:34.107573 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl" Oct 29 23:13:34.168146 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:34.168093 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w" podStartSLOduration=2.16807836 podStartE2EDuration="2.16807836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:34.163724237 +0000 UTC m=+1868.104272147" watchObservedRunningTime="2026-10-29 23:13:34.16807836 +0000 UTC m=+1868.108626267" Oct 29 23:13:34.211346 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:34.211298 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl" podStartSLOduration=2.21128353 podStartE2EDuration="2.21128353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:34.20578547 +0000 UTC m=+1868.146333381" watchObservedRunningTime="2026-10-29 23:13:34.21128353 +0000 UTC m=+1868.151831443" Oct 29 23:13:34.796610 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:34.796577 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 29 23:13:34.796882 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:34.796625 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" podUID="dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 29 23:13:35.796981 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:35.796940 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 29 23:13:35.797447 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:35.797004 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" podUID="dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 29 23:13:36.797486 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:36.797448 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 29 23:13:36.797917 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:36.797506 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" podUID="dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 29 23:13:37.532811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:37.532769 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:13:37.832785 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:37.832631 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:13:38.171901 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:38.171868 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:13:43.024273 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:43.024232 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be0401-882lr"] Oct 29 23:13:43.029172 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:43.029141 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be0401-882lr" Oct 29 23:13:43.051283 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:43.051248 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be0401-882lr"] Oct 29 23:13:43.061545 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:43.061515 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:13:43.061731 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:43.061713 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:13:43.141340 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:43.141305 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9930758c-47fa-42d6-a600-272488ea0128-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-be0401-882lr\" (UID: \"9930758c-47fa-42d6-a600-272488ea0128\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be0401-882lr" Oct 29 23:13:43.141532 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:43.141493 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm6nk\" (UniqueName: \"kubernetes.io/projected/9930758c-47fa-42d6-a600-272488ea0128-kube-api-access-bm6nk\") pod \"maas-api-revoke-keys-e2e-explicit-be0401-882lr\" (UID: \"9930758c-47fa-42d6-a600-272488ea0128\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be0401-882lr" Oct 29 23:13:43.141759 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:43.141741 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9930758c-47fa-42d6-a600-272488ea0128-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-be0401-882lr\" (UID: \"9930758c-47fa-42d6-a600-272488ea0128\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be0401-882lr" Oct 29 23:13:43.243187 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:43.243148 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9930758c-47fa-42d6-a600-272488ea0128-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-be0401-882lr\" (UID: \"9930758c-47fa-42d6-a600-272488ea0128\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be0401-882lr" Oct 29 23:13:43.243414 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:43.243328 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9930758c-47fa-42d6-a600-272488ea0128-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-be0401-882lr\" (UID: \"9930758c-47fa-42d6-a600-272488ea0128\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be0401-882lr" Oct 29 23:13:43.243498 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:43.243481 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm6nk\" (UniqueName: \"kubernetes.io/projected/9930758c-47fa-42d6-a600-272488ea0128-kube-api-access-bm6nk\") pod \"maas-api-revoke-keys-e2e-explicit-be0401-882lr\" (UID: \"9930758c-47fa-42d6-a600-272488ea0128\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be0401-882lr" Oct 29 23:13:43.243981 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:43.243961 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9930758c-47fa-42d6-a600-272488ea0128-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-be0401-882lr\" (UID: \"9930758c-47fa-42d6-a600-272488ea0128\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be0401-882lr" Oct 29 23:13:43.246164 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:43.246142 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9930758c-47fa-42d6-a600-272488ea0128-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-be0401-882lr\" (UID: \"9930758c-47fa-42d6-a600-272488ea0128\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be0401-882lr" Oct 29 23:13:43.268840 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:43.268809 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm6nk\" (UniqueName: \"kubernetes.io/projected/9930758c-47fa-42d6-a600-272488ea0128-kube-api-access-bm6nk\") pod \"maas-api-revoke-keys-e2e-explicit-be0401-882lr\" (UID: \"9930758c-47fa-42d6-a600-272488ea0128\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be0401-882lr" Oct 29 23:13:43.354451 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:43.354373 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be0401-882lr" Oct 29 23:13:43.669476 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:43.669444 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be0401-882lr"] Oct 29 23:13:43.669863 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:13:43.669833 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9930758c_47fa_42d6_a600_272488ea0128.slice/crio-53166552ebf69904cc1c1866dba4737414c3d04dedc9a69c57a0fbbf8f66faed WatchSource:0}: Error finding container 53166552ebf69904cc1c1866dba4737414c3d04dedc9a69c57a0fbbf8f66faed: Status 404 returned error can't find the container with id 53166552ebf69904cc1c1866dba4737414c3d04dedc9a69c57a0fbbf8f66faed Oct 29 23:13:44.035524 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:44.035498 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b37fde0-mvgkr"] Oct 29 23:13:44.046783 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:44.046756 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b37fde0-mvgkr"] Oct 29 23:13:44.183022 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:44.182925 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4adfb991-2490-439e-be0c-58c4ca399704" path="/var/lib/kubelet/pods/4adfb991-2490-439e-be0c-58c4ca399704/volumes" Oct 29 23:13:44.238126 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:44.238090 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be0401-882lr" event={"ID":"9930758c-47fa-42d6-a600-272488ea0128","Type":"ContainerStarted","Data":"b7ec4b890c128073f9d18dea98a7454270445657c6a48a8ddd84c2041021a75d"} Oct 29 23:13:44.238317 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:44.238147 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be0401-882lr" event={"ID":"9930758c-47fa-42d6-a600-272488ea0128","Type":"ContainerStarted","Data":"53166552ebf69904cc1c1866dba4737414c3d04dedc9a69c57a0fbbf8f66faed"} Oct 29 23:13:44.290923 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:44.290864 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be0401-882lr" podStartSLOduration=2.290848289 podStartE2EDuration="2.290848289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:44.289150562 +0000 UTC m=+1878.229698470" watchObservedRunningTime="2026-10-29 23:13:44.290848289 +0000 UTC m=+1878.231396245" Oct 29 23:13:45.127584 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:45.127555 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl" Oct 29 23:13:45.128242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:45.128189 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w" Oct 29 23:13:45.254633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:45.254593 4821 generic.go:356] "Generic (PLEG): container finished" podID="9930758c-47fa-42d6-a600-272488ea0128" containerID="b7ec4b890c128073f9d18dea98a7454270445657c6a48a8ddd84c2041021a75d" exitCode=0 Oct 29 23:13:45.254797 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:45.254681 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be0401-882lr" event={"ID":"9930758c-47fa-42d6-a600-272488ea0128","Type":"ContainerDied","Data":"b7ec4b890c128073f9d18dea98a7454270445657c6a48a8ddd84c2041021a75d"} Oct 29 23:13:46.433760 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:46.433735 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be0401-882lr" Oct 29 23:13:46.606088 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:46.606009 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9930758c-47fa-42d6-a600-272488ea0128-kube-api-access-bm6nk\" (UniqueName: \"kubernetes.io/projected/9930758c-47fa-42d6-a600-272488ea0128-kube-api-access-bm6nk\") pod \"9930758c-47fa-42d6-a600-272488ea0128\" (UID: \"9930758c-47fa-42d6-a600-272488ea0128\") " Oct 29 23:13:46.606088 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:46.606062 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9930758c-47fa-42d6-a600-272488ea0128-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9930758c-47fa-42d6-a600-272488ea0128-maas-api-xks-ca\") pod \"9930758c-47fa-42d6-a600-272488ea0128\" (UID: \"9930758c-47fa-42d6-a600-272488ea0128\") " Oct 29 23:13:46.606336 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:46.606319 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9930758c-47fa-42d6-a600-272488ea0128-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9930758c-47fa-42d6-a600-272488ea0128-maas-api-openshift-ca\") pod \"9930758c-47fa-42d6-a600-272488ea0128\" (UID: \"9930758c-47fa-42d6-a600-272488ea0128\") " Oct 29 23:13:46.606699 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:46.606672 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9930758c-47fa-42d6-a600-272488ea0128-maas-api-openshift-ca" pod "9930758c-47fa-42d6-a600-272488ea0128" (UID: "9930758c-47fa-42d6-a600-272488ea0128"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:13:46.606818 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:46.606802 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9930758c-47fa-42d6-a600-272488ea0128-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:13:46.608983 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:46.608959 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9930758c-47fa-42d6-a600-272488ea0128-maas-api-xks-ca" pod "9930758c-47fa-42d6-a600-272488ea0128" (UID: "9930758c-47fa-42d6-a600-272488ea0128"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:13:46.609088 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:46.609037 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9930758c-47fa-42d6-a600-272488ea0128-kube-api-access-bm6nk" pod "9930758c-47fa-42d6-a600-272488ea0128" (UID: "9930758c-47fa-42d6-a600-272488ea0128"). InnerVolumeSpecName "kube-api-access-bm6nk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:13:46.708319 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:46.708285 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm6nk\" (UniqueName: \"kubernetes.io/projected/9930758c-47fa-42d6-a600-272488ea0128-kube-api-access-bm6nk\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:13:46.708319 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:46.708317 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9930758c-47fa-42d6-a600-272488ea0128-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:13:47.272718 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:47.272679 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be0401-882lr" event={"ID":"9930758c-47fa-42d6-a600-272488ea0128","Type":"ContainerDied","Data":"53166552ebf69904cc1c1866dba4737414c3d04dedc9a69c57a0fbbf8f66faed"} Oct 29 23:13:47.272718 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:47.272702 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be0401-882lr" Oct 29 23:13:47.272944 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:47.272729 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53166552ebf69904cc1c1866dba4737414c3d04dedc9a69c57a0fbbf8f66faed" Oct 29 23:13:48.052377 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:48.052346 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18fab4db-2cwnf"] Oct 29 23:13:48.065022 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:48.064996 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18fab4db-2cwnf"] Oct 29 23:13:48.172041 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:48.172011 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91de5578-2f94-4e6c-81e5-70529f2f19d1" path="/var/lib/kubelet/pods/91de5578-2f94-4e6c-81e5-70529f2f19d1/volumes" Oct 29 23:13:53.887799 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:13:53.887760 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:13:59.033085 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:59.033049 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3bfcdda-dlkrg"] Oct 29 23:13:59.046056 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:13:59.046024 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3bfcdda-dlkrg"] Oct 29 23:14:00.171299 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:00.171264 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13abc4a1-190c-49b8-9603-6f64c96f9f1c" path="/var/lib/kubelet/pods/13abc4a1-190c-49b8-9603-6f64c96f9f1c/volumes" Oct 29 23:14:07.767265 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:07.767177 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be0401-882lr"] Oct 29 23:14:07.778128 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:07.778095 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-be0401-882lr"] Oct 29 23:14:08.172019 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:08.171970 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9930758c-47fa-42d6-a600-272488ea0128" path="/var/lib/kubelet/pods/9930758c-47fa-42d6-a600-272488ea0128/volumes" Oct 29 23:14:08.835231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:08.835169 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg"] Oct 29 23:14:08.836965 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:08.836932 4821 cpu_manager.go:414] "containerMap: removing container" podUID="9930758c-47fa-42d6-a600-272488ea0128" containerName="revoke-keys" Oct 29 23:14:08.836965 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:08.836967 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9930758c-47fa-42d6-a600-272488ea0128" containerName="revoke-keys" Oct 29 23:14:08.837247 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:08.837185 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="9930758c-47fa-42d6-a600-272488ea0128" containerName="revoke-keys" Oct 29 23:14:08.842820 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:08.842797 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:08.864463 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:08.864426 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg"] Oct 29 23:14:08.882753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:08.878667 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-49d87f-openshift-default-dockercfg-xk5gw\"" Oct 29 23:14:08.921162 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:08.921127 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-credential-socket\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:08.921386 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:08.921178 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-workload-certs\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:08.921386 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:08.921282 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f275822e-4f33-4366-8be5-348dc70ebffd-istio-token\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:08.921386 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:08.921381 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-workload-socket\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:08.921535 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:08.921443 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-istio-data\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:08.921535 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:08.921484 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r76vc\" (UniqueName: \"kubernetes.io/projected/f275822e-4f33-4366-8be5-348dc70ebffd-kube-api-access-r76vc\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:08.921645 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:08.921542 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f275822e-4f33-4366-8be5-348dc70ebffd-istio-podinfo\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:08.921790 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:08.921771 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-istio-envoy\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:08.921890 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:08.921875 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f275822e-4f33-4366-8be5-348dc70ebffd-istiod-ca-cert\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:09.026621 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.026584 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f275822e-4f33-4366-8be5-348dc70ebffd-istiod-ca-cert\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:09.026953 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.026911 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-credential-socket\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:09.027150 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.027121 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-workload-certs\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:09.027316 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.027299 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f275822e-4f33-4366-8be5-348dc70ebffd-istio-token\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:09.027539 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.027523 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-workload-socket\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:09.027744 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.027727 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-istio-data\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:09.027887 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.027871 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r76vc\" (UniqueName: \"kubernetes.io/projected/f275822e-4f33-4366-8be5-348dc70ebffd-kube-api-access-r76vc\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:09.028095 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.028079 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f275822e-4f33-4366-8be5-348dc70ebffd-istio-podinfo\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:09.028529 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.028508 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-istio-envoy\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:09.029933 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.029897 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-istio-data\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:09.030307 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.030279 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-credential-socket\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:09.041861 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.030838 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-workload-socket\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:09.043996 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.043875 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-workload-certs\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:09.044432 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.044409 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f275822e-4f33-4366-8be5-348dc70ebffd-istiod-ca-cert\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:09.091112 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.090420 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f275822e-4f33-4366-8be5-348dc70ebffd-istio-podinfo\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:09.139318 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.139284 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-istio-envoy\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:09.140062 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.140036 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f275822e-4f33-4366-8be5-348dc70ebffd-istio-token\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:09.181227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.172544 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r76vc\" (UniqueName: \"kubernetes.io/projected/f275822e-4f33-4366-8be5-348dc70ebffd-kube-api-access-r76vc\") pod \"e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:09.203602 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.203560 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:09.701310 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.701275 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg"] Oct 29 23:14:09.701570 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:14:09.701547 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf275822e_4f33_4366_8be5_348dc70ebffd.slice/crio-aba5149732889a4a9e1aa53426fbf46357bd4e617ba527d000b8c8d388a7c5f9 WatchSource:0}: Error finding container aba5149732889a4a9e1aa53426fbf46357bd4e617ba527d000b8c8d388a7c5f9: Status 404 returned error can't find the container with id aba5149732889a4a9e1aa53426fbf46357bd4e617ba527d000b8c8d388a7c5f9 Oct 29 23:14:09.704756 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.704711 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:14:09.704857 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.704807 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:14:09.704911 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.704851 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:14:09.878767 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.878733 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc"] Oct 29 23:14:09.883690 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.883529 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc" Oct 29 23:14:09.901073 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.901035 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-5e52e0-serving-cert\"" Oct 29 23:14:09.909177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.909145 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc"] Oct 29 23:14:09.968284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.968024 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2hwx\" (UniqueName: \"kubernetes.io/projected/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-kube-api-access-n2hwx\") pod \"maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc\" (UID: \"7d7c0ca7-37fe-4a65-a09b-e127842adb0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc" Oct 29 23:14:09.968284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.968159 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc\" (UID: \"7d7c0ca7-37fe-4a65-a09b-e127842adb0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc" Oct 29 23:14:09.968284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.968265 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc\" (UID: \"7d7c0ca7-37fe-4a65-a09b-e127842adb0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc" Oct 29 23:14:09.968602 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:09.968394 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc\" (UID: \"7d7c0ca7-37fe-4a65-a09b-e127842adb0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc" Oct 29 23:14:10.070424 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:10.070375 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2hwx\" (UniqueName: \"kubernetes.io/projected/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-kube-api-access-n2hwx\") pod \"maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc\" (UID: \"7d7c0ca7-37fe-4a65-a09b-e127842adb0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc" Oct 29 23:14:10.070629 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:10.070455 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc\" (UID: \"7d7c0ca7-37fe-4a65-a09b-e127842adb0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc" Oct 29 23:14:10.070629 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:10.070540 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc\" (UID: \"7d7c0ca7-37fe-4a65-a09b-e127842adb0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc" Oct 29 23:14:10.070763 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:10.070681 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc\" (UID: \"7d7c0ca7-37fe-4a65-a09b-e127842adb0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc" Oct 29 23:14:10.071537 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:10.071508 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc\" (UID: \"7d7c0ca7-37fe-4a65-a09b-e127842adb0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc" Oct 29 23:14:10.073735 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:10.073644 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc\" (UID: \"7d7c0ca7-37fe-4a65-a09b-e127842adb0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc" Oct 29 23:14:10.074605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:10.074583 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc\" (UID: \"7d7c0ca7-37fe-4a65-a09b-e127842adb0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc" Oct 29 23:14:10.098184 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:10.098148 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2hwx\" (UniqueName: \"kubernetes.io/projected/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-kube-api-access-n2hwx\") pod \"maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc\" (UID: \"7d7c0ca7-37fe-4a65-a09b-e127842adb0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc" Oct 29 23:14:10.205216 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:10.204310 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc" Oct 29 23:14:10.518374 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:10.518283 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" event={"ID":"f275822e-4f33-4366-8be5-348dc70ebffd","Type":"ContainerStarted","Data":"19e90a2a30dd1b2974e6f079979fea2d04f67623ea7fe4eafb908ebb571244b9"} Oct 29 23:14:10.518374 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:10.518334 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" event={"ID":"f275822e-4f33-4366-8be5-348dc70ebffd","Type":"ContainerStarted","Data":"aba5149732889a4a9e1aa53426fbf46357bd4e617ba527d000b8c8d388a7c5f9"} Oct 29 23:14:10.568322 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:10.568261 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" podStartSLOduration=2.568241845 podStartE2EDuration="2.568241845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:10.564794597 +0000 UTC m=+1904.505342506" watchObservedRunningTime="2026-10-29 23:14:10.568241845 +0000 UTC m=+1904.508789754" Oct 29 23:14:10.602030 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:10.601994 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc"] Oct 29 23:14:11.212227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:11.211017 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:11.348283 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:11.347547 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:11.528714 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:11.528670 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc" event={"ID":"7d7c0ca7-37fe-4a65-a09b-e127842adb0b","Type":"ContainerStarted","Data":"825eb47b71364548d39015ee48f5fc79f2003bc6690339f240fc06c8b3b00cff"} Oct 29 23:14:11.528912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:11.528773 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc" event={"ID":"7d7c0ca7-37fe-4a65-a09b-e127842adb0b","Type":"ContainerStarted","Data":"91166794adca5b90605ef6c4c27593be76a020795458dd9d77ea0ee17035c803"} Oct 29 23:14:11.529003 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:11.528927 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc" Oct 29 23:14:11.529277 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:11.529260 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:11.530246 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:11.530192 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:14:11.573325 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:11.573259 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc" podStartSLOduration=2.108036076 podStartE2EDuration="2.573241623s" totalImagesPullingTime="465.205547ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:11.572523987 +0000 UTC m=+1905.513071898" watchObservedRunningTime="2026-10-29 23:14:11.573241623 +0000 UTC m=+1905.513789536" Oct 29 23:14:17.544702 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:17.544667 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc" Oct 29 23:14:23.887239 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:14:23.887170 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:14:29.428178 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:29.428140 4821 scope.go:122] "RemoveContainer" containerID="30ffe8d1edb3718b7d6c7e81177a71c47663218703e7b6ff39c3d410a6852e10" Oct 29 23:14:29.439830 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:29.439809 4821 scope.go:122] "RemoveContainer" containerID="b418af0f31a65920f450e2c7d16201ea382144c97ab3b234632a12800fff8704" Oct 29 23:14:29.452434 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:29.452417 4821 scope.go:122] "RemoveContainer" containerID="410ecacab497ead8a0fd84a38d3880fdb46ceaf99c25f80661434fa0a5ec347b" Oct 29 23:14:32.981005 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:32.980969 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw"] Oct 29 23:14:32.987126 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:32.987103 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.010289 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.010262 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-4504f1-openshift-default-dockercfg-28rzf\"" Oct 29 23:14:33.021894 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.021846 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw"] Oct 29 23:14:33.066255 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.064422 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f36035d-zft54"] Oct 29 23:14:33.071269 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.071191 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f36035d-zft54"] Oct 29 23:14:33.095056 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.095022 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073c5be5-712e-4634-82a1-9f957244d5e1-istio-podinfo\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.095305 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.095078 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-istio-envoy\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.095305 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.095227 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27gcp\" (UniqueName: \"kubernetes.io/projected/073c5be5-712e-4634-82a1-9f957244d5e1-kube-api-access-27gcp\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.095305 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.095285 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-workload-socket\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.095305 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.095306 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/073c5be5-712e-4634-82a1-9f957244d5e1-istio-token\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.095607 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.095340 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-credential-socket\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.095607 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.095359 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-istio-data\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.095607 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.095490 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-workload-certs\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.095607 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.095541 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073c5be5-712e-4634-82a1-9f957244d5e1-istiod-ca-cert\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.196749 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.196719 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-workload-certs\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.196928 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.196785 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073c5be5-712e-4634-82a1-9f957244d5e1-istiod-ca-cert\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.196928 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.196876 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073c5be5-712e-4634-82a1-9f957244d5e1-istio-podinfo\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.196928 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.196911 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-istio-envoy\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.197106 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.197067 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27gcp\" (UniqueName: \"kubernetes.io/projected/073c5be5-712e-4634-82a1-9f957244d5e1-kube-api-access-27gcp\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.197183 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.197127 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-workload-socket\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.197183 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.197160 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/073c5be5-712e-4634-82a1-9f957244d5e1-istio-token\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.197384 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.197179 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-workload-certs\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.197384 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.197256 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-credential-socket\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.197384 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.197284 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-istio-data\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.197556 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.197475 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-workload-socket\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.197619 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.197580 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-credential-socket\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.197845 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.197639 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-istio-data\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.197845 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.197675 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073c5be5-712e-4634-82a1-9f957244d5e1-istiod-ca-cert\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.199401 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.199377 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-istio-envoy\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.199638 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.199620 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073c5be5-712e-4634-82a1-9f957244d5e1-istio-podinfo\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.221585 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.221556 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/073c5be5-712e-4634-82a1-9f957244d5e1-istio-token\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.221677 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.221661 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27gcp\" (UniqueName: \"kubernetes.io/projected/073c5be5-712e-4634-82a1-9f957244d5e1-kube-api-access-27gcp\") pod \"e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.302522 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.302447 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:33.632010 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:14:33.631972 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod073c5be5_712e_4634_82a1_9f957244d5e1.slice/crio-8b436f0bef497dd483d313f2816a113eba6a6860f6c52c4bddce8e6c6a50d806 WatchSource:0}: Error finding container 8b436f0bef497dd483d313f2816a113eba6a6860f6c52c4bddce8e6c6a50d806: Status 404 returned error can't find the container with id 8b436f0bef497dd483d313f2816a113eba6a6860f6c52c4bddce8e6c6a50d806 Oct 29 23:14:33.632660 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.632638 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw"] Oct 29 23:14:33.634904 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.634869 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:14:33.634998 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.634951 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:14:33.634998 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.634992 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:14:33.747656 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.747621 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" event={"ID":"073c5be5-712e-4634-82a1-9f957244d5e1","Type":"ContainerStarted","Data":"be155f2bf566b59089ba78a18a5fb3dc25269005ce0063e5b783a15127bf8e6d"} Oct 29 23:14:33.747841 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.747679 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" event={"ID":"073c5be5-712e-4634-82a1-9f957244d5e1","Type":"ContainerStarted","Data":"8b436f0bef497dd483d313f2816a113eba6a6860f6c52c4bddce8e6c6a50d806"} Oct 29 23:14:33.803786 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:33.803727 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" podStartSLOduration=1.803707353 podStartE2EDuration="1.803707353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:33.797809657 +0000 UTC m=+1927.738357576" watchObservedRunningTime="2026-10-29 23:14:33.803707353 +0000 UTC m=+1927.744255267" Oct 29 23:14:34.173452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:34.173423 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7d413f3-e64a-4367-948a-5d96ff9775de" path="/var/lib/kubelet/pods/d7d413f3-e64a-4367-948a-5d96ff9775de/volumes" Oct 29 23:14:34.303099 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:34.303065 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:34.304934 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:34.304912 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 29 23:14:34.305023 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:34.304971 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" podUID="073c5be5-712e-4634-82a1-9f957244d5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 29 23:14:35.303178 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:35.303135 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 29 23:14:35.303637 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:35.303192 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" podUID="073c5be5-712e-4634-82a1-9f957244d5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 29 23:14:36.303746 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:36.303709 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 29 23:14:36.304265 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:36.303774 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" podUID="073c5be5-712e-4634-82a1-9f957244d5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 29 23:14:37.302963 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:37.302875 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 29 23:14:37.302963 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:37.302945 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" podUID="073c5be5-712e-4634-82a1-9f957244d5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 29 23:14:38.303600 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:38.303563 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 29 23:14:38.304015 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:38.303629 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" podUID="073c5be5-712e-4634-82a1-9f957244d5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 29 23:14:39.303761 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:39.303728 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 29 23:14:39.304317 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:39.303782 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" podUID="073c5be5-712e-4634-82a1-9f957244d5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 29 23:14:40.303597 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:40.303564 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 29 23:14:40.303788 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:40.303615 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" podUID="073c5be5-712e-4634-82a1-9f957244d5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 29 23:14:41.303029 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:41.302992 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 29 23:14:41.303262 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:41.303051 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" podUID="073c5be5-712e-4634-82a1-9f957244d5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 29 23:14:42.307828 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:42.307798 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:42.308317 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:42.308297 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:42.308958 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:42.308938 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:14:45.032315 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:45.032278 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a004662-7nj9d"] Oct 29 23:14:45.046530 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:45.046500 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a004662-7nj9d"] Oct 29 23:14:46.172463 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:46.172427 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d7b420c-09a6-4125-82df-699549ab29ee" path="/var/lib/kubelet/pods/9d7b420c-09a6-4125-82df-699549ab29ee/volumes" Oct 29 23:14:48.645995 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:48.645964 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988"] Oct 29 23:14:48.646513 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:48.646154 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" podUID="e5ffde39-a456-47e3-8fea-4ce3f51084b3" containerName="istio-proxy" containerID="cri-o://a918a606c674fddd9381cd22be767cece4f4cc31b42ccb7914b0704f967612d5" gracePeriod=30 Oct 29 23:14:48.661307 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:48.661272 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp"] Oct 29 23:14:48.661522 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:48.661494 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" podUID="23d4c06e-ed49-45da-85d5-55e60a3d48c1" containerName="istio-proxy" containerID="cri-o://2624cf98aad5c6b0206f8aed0e6085196b60a46c21174d632fdcdf6d8bb53e0b" gracePeriod=30 Oct 29 23:14:52.686312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:52.686262 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:14:52.686865 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:52.686384 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" podUID="23d4c06e-ed49-45da-85d5-55e60a3d48c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:14:53.759066 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:53.759039 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43add4fd-dnk8c"] Oct 29 23:14:53.763423 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:53.763405 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43add4fd-dnk8c" Oct 29 23:14:53.779561 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:53.779528 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:14:53.785281 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:53.785253 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43add4fd-dnk8c"] Oct 29 23:14:53.800331 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:53.799472 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:14:53.880771 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:53.880735 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82cd32c0-9b49-4b74-8d05-d64ee8576c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43add4fd-dnk8c\" (UID: \"82cd32c0-9b49-4b74-8d05-d64ee8576c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43add4fd-dnk8c" Oct 29 23:14:53.881347 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:53.881325 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tljtw\" (UniqueName: \"kubernetes.io/projected/82cd32c0-9b49-4b74-8d05-d64ee8576c8d-kube-api-access-tljtw\") pod \"maas-api-revoke-sub-43add4fd-dnk8c\" (UID: \"82cd32c0-9b49-4b74-8d05-d64ee8576c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43add4fd-dnk8c" Oct 29 23:14:53.881671 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:53.881654 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82cd32c0-9b49-4b74-8d05-d64ee8576c8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43add4fd-dnk8c\" (UID: \"82cd32c0-9b49-4b74-8d05-d64ee8576c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43add4fd-dnk8c" Oct 29 23:14:53.907591 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:14:53.907545 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:14:53.984585 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:53.984544 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tljtw\" (UniqueName: \"kubernetes.io/projected/82cd32c0-9b49-4b74-8d05-d64ee8576c8d-kube-api-access-tljtw\") pod \"maas-api-revoke-sub-43add4fd-dnk8c\" (UID: \"82cd32c0-9b49-4b74-8d05-d64ee8576c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43add4fd-dnk8c" Oct 29 23:14:53.984768 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:53.984684 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82cd32c0-9b49-4b74-8d05-d64ee8576c8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43add4fd-dnk8c\" (UID: \"82cd32c0-9b49-4b74-8d05-d64ee8576c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43add4fd-dnk8c" Oct 29 23:14:53.985373 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:53.984911 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82cd32c0-9b49-4b74-8d05-d64ee8576c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43add4fd-dnk8c\" (UID: \"82cd32c0-9b49-4b74-8d05-d64ee8576c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43add4fd-dnk8c" Oct 29 23:14:53.986622 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:53.985946 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82cd32c0-9b49-4b74-8d05-d64ee8576c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43add4fd-dnk8c\" (UID: \"82cd32c0-9b49-4b74-8d05-d64ee8576c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43add4fd-dnk8c" Oct 29 23:14:54.044259 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.041506 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82cd32c0-9b49-4b74-8d05-d64ee8576c8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43add4fd-dnk8c\" (UID: \"82cd32c0-9b49-4b74-8d05-d64ee8576c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43add4fd-dnk8c" Oct 29 23:14:54.359519 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.358737 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tljtw\" (UniqueName: \"kubernetes.io/projected/82cd32c0-9b49-4b74-8d05-d64ee8576c8d-kube-api-access-tljtw\") pod \"maas-api-revoke-sub-43add4fd-dnk8c\" (UID: \"82cd32c0-9b49-4b74-8d05-d64ee8576c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43add4fd-dnk8c" Oct 29 23:14:54.421164 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.419571 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43add4fd-dnk8c" Oct 29 23:14:54.430340 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.430004 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 23:14:54.449665 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.449633 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:14:54.453600 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.453378 4821 generic.go:356] "Generic (PLEG): container finished" podID="e5ffde39-a456-47e3-8fea-4ce3f51084b3" containerID="a918a606c674fddd9381cd22be767cece4f4cc31b42ccb7914b0704f967612d5" exitCode=0 Oct 29 23:14:54.453600 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.453456 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" event={"ID":"e5ffde39-a456-47e3-8fea-4ce3f51084b3","Type":"ContainerDied","Data":"a918a606c674fddd9381cd22be767cece4f4cc31b42ccb7914b0704f967612d5"} Oct 29 23:14:54.453600 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.453495 4821 scope.go:122] "RemoveContainer" containerID="a918a606c674fddd9381cd22be767cece4f4cc31b42ccb7914b0704f967612d5" Oct 29 23:14:54.460039 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.456821 4821 generic.go:356] "Generic (PLEG): container finished" podID="23d4c06e-ed49-45da-85d5-55e60a3d48c1" containerID="2624cf98aad5c6b0206f8aed0e6085196b60a46c21174d632fdcdf6d8bb53e0b" exitCode=0 Oct 29 23:14:54.460039 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.459333 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" Oct 29 23:14:54.460039 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.459568 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp" event={"ID":"23d4c06e-ed49-45da-85d5-55e60a3d48c1","Type":"ContainerDied","Data":"2624cf98aad5c6b0206f8aed0e6085196b60a46c21174d632fdcdf6d8bb53e0b"} Oct 29 23:14:54.483059 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.482855 4821 scope.go:122] "RemoveContainer" containerID="2624cf98aad5c6b0206f8aed0e6085196b60a46c21174d632fdcdf6d8bb53e0b" Oct 29 23:14:54.498833 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.498801 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-podinfo\") pod \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " Oct 29 23:14:54.499512 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.499489 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ffde39-a456-47e3-8fea-4ce3f51084b3-kube-api-access-k6chk\" (UniqueName: \"kubernetes.io/projected/e5ffde39-a456-47e3-8fea-4ce3f51084b3-kube-api-access-k6chk\") pod \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " Oct 29 23:14:54.509746 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.509703 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ffde39-a456-47e3-8fea-4ce3f51084b3-kube-api-access-k6chk" pod "e5ffde39-a456-47e3-8fea-4ce3f51084b3" (UID: "e5ffde39-a456-47e3-8fea-4ce3f51084b3"). InnerVolumeSpecName "kube-api-access-k6chk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:14:54.513218 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.513167 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-podinfo" pod "e5ffde39-a456-47e3-8fea-4ce3f51084b3" (UID: "e5ffde39-a456-47e3-8fea-4ce3f51084b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:14:54.607966 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.607917 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-data\") pod \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " Oct 29 23:14:54.608318 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.608276 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-workload-socket\") pod \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " Oct 29 23:14:54.608944 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.608511 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-envoy\") pod \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " Oct 29 23:14:54.609180 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.608754 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-workload-socket" pod "e5ffde39-a456-47e3-8fea-4ce3f51084b3" (UID: "e5ffde39-a456-47e3-8fea-4ce3f51084b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:14:54.609414 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.609399 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-workload-socket\") pod \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " Oct 29 23:14:54.610081 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.609723 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-workload-socket" pod "23d4c06e-ed49-45da-85d5-55e60a3d48c1" (UID: "23d4c06e-ed49-45da-85d5-55e60a3d48c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:14:54.610081 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.609505 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-data" pod "e5ffde39-a456-47e3-8fea-4ce3f51084b3" (UID: "e5ffde39-a456-47e3-8fea-4ce3f51084b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:14:54.610725 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.610685 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istiod-ca-cert" pod "23d4c06e-ed49-45da-85d5-55e60a3d48c1" (UID: "23d4c06e-ed49-45da-85d5-55e60a3d48c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:14:54.610883 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.610865 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istiod-ca-cert\") pod \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " Oct 29 23:14:54.611053 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.611028 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-credential-socket\") pod \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " Oct 29 23:14:54.611768 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.611535 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-workload-certs\") pod \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " Oct 29 23:14:54.612062 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.611430 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-credential-socket" pod "23d4c06e-ed49-45da-85d5-55e60a3d48c1" (UID: "23d4c06e-ed49-45da-85d5-55e60a3d48c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:14:54.612062 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.611741 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-workload-certs" pod "e5ffde39-a456-47e3-8fea-4ce3f51084b3" (UID: "e5ffde39-a456-47e3-8fea-4ce3f51084b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:14:54.612550 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.612009 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-podinfo\") pod \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " Oct 29 23:14:54.613191 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.613157 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istiod-ca-cert" pod "e5ffde39-a456-47e3-8fea-4ce3f51084b3" (UID: "e5ffde39-a456-47e3-8fea-4ce3f51084b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:14:54.613369 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.613352 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istiod-ca-cert\") pod \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " Oct 29 23:14:54.613632 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.613598 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-envoy\") pod \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " Oct 29 23:14:54.613811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.613788 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-token\" (UniqueName: \"kubernetes.io/projected/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-token\") pod \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " Oct 29 23:14:54.613978 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.613960 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-data\") pod \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " Oct 29 23:14:54.614160 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.614126 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d4c06e-ed49-45da-85d5-55e60a3d48c1-kube-api-access-vpcnw\" (UniqueName: \"kubernetes.io/projected/23d4c06e-ed49-45da-85d5-55e60a3d48c1-kube-api-access-vpcnw\") pod \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " Oct 29 23:14:54.617412 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.617393 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-workload-certs\") pod \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\" (UID: \"23d4c06e-ed49-45da-85d5-55e60a3d48c1\") " Oct 29 23:14:54.617570 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.617555 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-token\" (UniqueName: \"kubernetes.io/projected/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-token\") pod \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " Oct 29 23:14:54.617706 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.617690 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-credential-socket\") pod \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\" (UID: \"e5ffde39-a456-47e3-8fea-4ce3f51084b3\") " Oct 29 23:14:54.619029 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.616714 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-envoy" pod "e5ffde39-a456-47e3-8fea-4ce3f51084b3" (UID: "e5ffde39-a456-47e3-8fea-4ce3f51084b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:14:54.619029 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.618392 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-data" pod "23d4c06e-ed49-45da-85d5-55e60a3d48c1" (UID: "23d4c06e-ed49-45da-85d5-55e60a3d48c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:14:54.619029 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.618989 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-workload-certs" pod "23d4c06e-ed49-45da-85d5-55e60a3d48c1" (UID: "23d4c06e-ed49-45da-85d5-55e60a3d48c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:14:54.619624 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.619566 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-credential-socket" pod "e5ffde39-a456-47e3-8fea-4ce3f51084b3" (UID: "e5ffde39-a456-47e3-8fea-4ce3f51084b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:14:54.624642 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.624517 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-envoy" pod "23d4c06e-ed49-45da-85d5-55e60a3d48c1" (UID: "23d4c06e-ed49-45da-85d5-55e60a3d48c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:14:54.626977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.626945 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:14:54.627345 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.627323 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:14:54.627464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.627357 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6chk\" (UniqueName: \"kubernetes.io/projected/e5ffde39-a456-47e3-8fea-4ce3f51084b3-kube-api-access-k6chk\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:14:54.627464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.627372 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:14:54.627464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.627385 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:14:54.627464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.627398 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:14:54.627464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.627414 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:14:54.627464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.627427 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:14:54.627464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.627440 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:14:54.627464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.627459 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:14:54.627889 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.627475 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:14:54.627889 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.627488 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:14:54.627889 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.627502 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23d4c06e-ed49-45da-85d5-55e60a3d48c1-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:14:54.627889 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.627516 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ffde39-a456-47e3-8fea-4ce3f51084b3-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:14:54.630592 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.630547 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-podinfo" pod "23d4c06e-ed49-45da-85d5-55e60a3d48c1" (UID: "23d4c06e-ed49-45da-85d5-55e60a3d48c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:14:54.631065 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.630894 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d4c06e-ed49-45da-85d5-55e60a3d48c1-kube-api-access-vpcnw" pod "23d4c06e-ed49-45da-85d5-55e60a3d48c1" (UID: "23d4c06e-ed49-45da-85d5-55e60a3d48c1"). InnerVolumeSpecName "kube-api-access-vpcnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:14:54.631065 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.631025 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-token" pod "23d4c06e-ed49-45da-85d5-55e60a3d48c1" (UID: "23d4c06e-ed49-45da-85d5-55e60a3d48c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:14:54.631420 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.631087 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-token" pod "e5ffde39-a456-47e3-8fea-4ce3f51084b3" (UID: "e5ffde39-a456-47e3-8fea-4ce3f51084b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:14:54.743906 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.734316 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:14:54.743906 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.734352 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23d4c06e-ed49-45da-85d5-55e60a3d48c1-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:14:54.743906 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.734367 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpcnw\" (UniqueName: \"kubernetes.io/projected/23d4c06e-ed49-45da-85d5-55e60a3d48c1-kube-api-access-vpcnw\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:14:54.743906 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.734383 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ffde39-a456-47e3-8fea-4ce3f51084b3-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:14:54.822846 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.822800 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp"] Oct 29 23:14:54.828437 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.828404 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-s94qp"] Oct 29 23:14:54.880271 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:54.878725 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43add4fd-dnk8c"] Oct 29 23:14:54.880697 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:14:54.880595 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82cd32c0_9b49_4b74_8d05_d64ee8576c8d.slice/crio-8b85b1ef3523e34419face26a960d54307c5a1572d0ad08703552ee5c857d691 WatchSource:0}: Error finding container 8b85b1ef3523e34419face26a960d54307c5a1572d0ad08703552ee5c857d691: Status 404 returned error can't find the container with id 8b85b1ef3523e34419face26a960d54307c5a1572d0ad08703552ee5c857d691 Oct 29 23:14:55.472884 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:55.472837 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43add4fd-dnk8c" event={"ID":"82cd32c0-9b49-4b74-8d05-d64ee8576c8d","Type":"ContainerStarted","Data":"f27e133cecacd9a539a2648aa5caf3d2e2b58d1f047ed238d8b062904f4caca7"} Oct 29 23:14:55.473072 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:55.472905 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43add4fd-dnk8c" event={"ID":"82cd32c0-9b49-4b74-8d05-d64ee8576c8d","Type":"ContainerStarted","Data":"8b85b1ef3523e34419face26a960d54307c5a1572d0ad08703552ee5c857d691"} Oct 29 23:14:55.475106 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:55.475067 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" Oct 29 23:14:55.475315 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:55.475104 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988" event={"ID":"e5ffde39-a456-47e3-8fea-4ce3f51084b3","Type":"ContainerDied","Data":"2eb06ea3d2bd325694496d596293469a9039d4ba5f8fbf776172b35ec7ed6074"} Oct 29 23:14:55.531575 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:55.531463 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43add4fd-dnk8c" podStartSLOduration=2.531427368 podStartE2EDuration="2.531427368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:55.528082131 +0000 UTC m=+1949.468630065" watchObservedRunningTime="2026-10-29 23:14:55.531427368 +0000 UTC m=+1949.471975279" Oct 29 23:14:55.575509 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:55.575463 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988"] Oct 29 23:14:55.581464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:55.581433 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-qp988"] Oct 29 23:14:56.202404 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:56.202366 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d4c06e-ed49-45da-85d5-55e60a3d48c1" path="/var/lib/kubelet/pods/23d4c06e-ed49-45da-85d5-55e60a3d48c1/volumes" Oct 29 23:14:56.203483 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:56.203169 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ffde39-a456-47e3-8fea-4ce3f51084b3" path="/var/lib/kubelet/pods/e5ffde39-a456-47e3-8fea-4ce3f51084b3/volumes" Oct 29 23:14:56.241349 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:56.241312 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7605-61b098"] Oct 29 23:14:56.242343 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:56.242317 4821 cpu_manager.go:414] "containerMap: removing container" podUID="23d4c06e-ed49-45da-85d5-55e60a3d48c1" containerName="istio-proxy" Oct 29 23:14:56.242343 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:56.242340 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d4c06e-ed49-45da-85d5-55e60a3d48c1" containerName="istio-proxy" Oct 29 23:14:56.242564 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:56.242355 4821 cpu_manager.go:414] "containerMap: removing container" podUID="e5ffde39-a456-47e3-8fea-4ce3f51084b3" containerName="istio-proxy" Oct 29 23:14:56.242564 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:56.242360 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ffde39-a456-47e3-8fea-4ce3f51084b3" containerName="istio-proxy" Oct 29 23:14:56.242564 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:56.242516 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d4c06e-ed49-45da-85d5-55e60a3d48c1" containerName="istio-proxy" Oct 29 23:14:56.242564 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:56.242531 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ffde39-a456-47e3-8fea-4ce3f51084b3" containerName="istio-proxy" Oct 29 23:14:56.246593 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:56.246300 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7605-61b098" Oct 29 23:14:56.262587 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:56.262562 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fvr2l\"" Oct 29 23:14:56.268794 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:56.268764 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7605-61b098"] Oct 29 23:14:56.385404 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:56.385359 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8xsz\" (UniqueName: \"kubernetes.io/projected/c20ecb28-30a7-4c78-bfd4-8c0896baf52f-kube-api-access-d8xsz\") pod \"test-curl-7605-61b098\" (UID: \"c20ecb28-30a7-4c78-bfd4-8c0896baf52f\") " pod="opendatahub/test-curl-7605-61b098" Oct 29 23:14:56.488434 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:56.488346 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8xsz\" (UniqueName: \"kubernetes.io/projected/c20ecb28-30a7-4c78-bfd4-8c0896baf52f-kube-api-access-d8xsz\") pod \"test-curl-7605-61b098\" (UID: \"c20ecb28-30a7-4c78-bfd4-8c0896baf52f\") " pod="opendatahub/test-curl-7605-61b098" Oct 29 23:14:56.501731 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:56.501699 4821 generic.go:356] "Generic (PLEG): container finished" podID="82cd32c0-9b49-4b74-8d05-d64ee8576c8d" containerID="f27e133cecacd9a539a2648aa5caf3d2e2b58d1f047ed238d8b062904f4caca7" exitCode=0 Oct 29 23:14:56.501912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:56.501855 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43add4fd-dnk8c" event={"ID":"82cd32c0-9b49-4b74-8d05-d64ee8576c8d","Type":"ContainerDied","Data":"f27e133cecacd9a539a2648aa5caf3d2e2b58d1f047ed238d8b062904f4caca7"} Oct 29 23:14:56.520603 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:56.520559 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8xsz\" (UniqueName: \"kubernetes.io/projected/c20ecb28-30a7-4c78-bfd4-8c0896baf52f-kube-api-access-d8xsz\") pod \"test-curl-7605-61b098\" (UID: \"c20ecb28-30a7-4c78-bfd4-8c0896baf52f\") " pod="opendatahub/test-curl-7605-61b098" Oct 29 23:14:56.601097 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:56.601049 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7605-61b098" Oct 29 23:14:56.985349 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:14:56.985321 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc20ecb28_30a7_4c78_bfd4_8c0896baf52f.slice/crio-f053b3ada2c82808ab9938a38278d1bd30b43e8baee2df9bc60bb46aba3f793e WatchSource:0}: Error finding container f053b3ada2c82808ab9938a38278d1bd30b43e8baee2df9bc60bb46aba3f793e: Status 404 returned error can't find the container with id f053b3ada2c82808ab9938a38278d1bd30b43e8baee2df9bc60bb46aba3f793e Oct 29 23:14:56.985815 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:56.985788 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7605-61b098"] Oct 29 23:14:57.512360 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:57.512322 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7605-61b098" event={"ID":"c20ecb28-30a7-4c78-bfd4-8c0896baf52f","Type":"ContainerStarted","Data":"7445d4f91338a62208c881ea6865d01c95e56dce6365c4dca7c0ef8e4c76d7e6"} Oct 29 23:14:57.512360 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:57.512366 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7605-61b098" event={"ID":"c20ecb28-30a7-4c78-bfd4-8c0896baf52f","Type":"ContainerStarted","Data":"f053b3ada2c82808ab9938a38278d1bd30b43e8baee2df9bc60bb46aba3f793e"} Oct 29 23:14:57.555643 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:57.555589 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7605-61b098" podStartSLOduration=1.555574748 podStartE2EDuration="1.555574748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:57.554358947 +0000 UTC m=+1951.494906859" watchObservedRunningTime="2026-10-29 23:14:57.555574748 +0000 UTC m=+1951.496122652" Oct 29 23:14:57.702309 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:57.702279 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43add4fd-dnk8c" Oct 29 23:14:57.816467 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:57.816383 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82cd32c0-9b49-4b74-8d05-d64ee8576c8d-kube-api-access-tljtw\" (UniqueName: \"kubernetes.io/projected/82cd32c0-9b49-4b74-8d05-d64ee8576c8d-kube-api-access-tljtw\") pod \"82cd32c0-9b49-4b74-8d05-d64ee8576c8d\" (UID: \"82cd32c0-9b49-4b74-8d05-d64ee8576c8d\") " Oct 29 23:14:57.816637 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:57.816534 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82cd32c0-9b49-4b74-8d05-d64ee8576c8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82cd32c0-9b49-4b74-8d05-d64ee8576c8d-maas-api-openshift-ca\") pod \"82cd32c0-9b49-4b74-8d05-d64ee8576c8d\" (UID: \"82cd32c0-9b49-4b74-8d05-d64ee8576c8d\") " Oct 29 23:14:57.816637 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:57.816563 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82cd32c0-9b49-4b74-8d05-d64ee8576c8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82cd32c0-9b49-4b74-8d05-d64ee8576c8d-maas-api-xks-ca\") pod \"82cd32c0-9b49-4b74-8d05-d64ee8576c8d\" (UID: \"82cd32c0-9b49-4b74-8d05-d64ee8576c8d\") " Oct 29 23:14:57.816887 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:57.816857 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82cd32c0-9b49-4b74-8d05-d64ee8576c8d-maas-api-openshift-ca" pod "82cd32c0-9b49-4b74-8d05-d64ee8576c8d" (UID: "82cd32c0-9b49-4b74-8d05-d64ee8576c8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:14:57.817362 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:57.817284 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82cd32c0-9b49-4b74-8d05-d64ee8576c8d-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:14:57.818784 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:57.818760 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82cd32c0-9b49-4b74-8d05-d64ee8576c8d-maas-api-xks-ca" pod "82cd32c0-9b49-4b74-8d05-d64ee8576c8d" (UID: "82cd32c0-9b49-4b74-8d05-d64ee8576c8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:14:57.818942 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:57.818922 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82cd32c0-9b49-4b74-8d05-d64ee8576c8d-kube-api-access-tljtw" pod "82cd32c0-9b49-4b74-8d05-d64ee8576c8d" (UID: "82cd32c0-9b49-4b74-8d05-d64ee8576c8d"). InnerVolumeSpecName "kube-api-access-tljtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:14:57.918786 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:57.918755 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tljtw\" (UniqueName: \"kubernetes.io/projected/82cd32c0-9b49-4b74-8d05-d64ee8576c8d-kube-api-access-tljtw\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:14:57.918786 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:57.918786 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82cd32c0-9b49-4b74-8d05-d64ee8576c8d-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:14:58.456574 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.456491 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7605-61b098"] Oct 29 23:14:58.461520 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.461493 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7605-61b098"] Oct 29 23:14:58.523887 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.523859 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43add4fd-dnk8c" Oct 29 23:14:58.524296 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.523884 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43add4fd-dnk8c" event={"ID":"82cd32c0-9b49-4b74-8d05-d64ee8576c8d","Type":"ContainerDied","Data":"8b85b1ef3523e34419face26a960d54307c5a1572d0ad08703552ee5c857d691"} Oct 29 23:14:58.524296 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.523937 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b85b1ef3523e34419face26a960d54307c5a1572d0ad08703552ee5c857d691" Oct 29 23:14:58.658788 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.658755 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46d37e92-phvgz"] Oct 29 23:14:58.659505 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.659486 4821 cpu_manager.go:414] "containerMap: removing container" podUID="c20ecb28-30a7-4c78-bfd4-8c0896baf52f" containerName="test-curl-7605-61b098" Oct 29 23:14:58.659581 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.659506 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c20ecb28-30a7-4c78-bfd4-8c0896baf52f" containerName="test-curl-7605-61b098" Oct 29 23:14:58.659581 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.659560 4821 cpu_manager.go:414] "containerMap: removing container" podUID="82cd32c0-9b49-4b74-8d05-d64ee8576c8d" containerName="revoke-keys" Oct 29 23:14:58.659581 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.659565 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82cd32c0-9b49-4b74-8d05-d64ee8576c8d" containerName="revoke-keys" Oct 29 23:14:58.659686 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.659677 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="82cd32c0-9b49-4b74-8d05-d64ee8576c8d" containerName="revoke-keys" Oct 29 23:14:58.659723 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.659690 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="c20ecb28-30a7-4c78-bfd4-8c0896baf52f" containerName="test-curl-7605-61b098" Oct 29 23:14:58.669739 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.669718 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d37e92-phvgz" Oct 29 23:14:58.689166 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.689072 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46d37e92-phvgz"] Oct 29 23:14:58.690110 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.690083 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:14:58.691532 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.691509 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:14:58.761655 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.761581 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sx97\" (UniqueName: \"kubernetes.io/projected/def6461d-2480-45dc-b7cd-9f7047855ca7-kube-api-access-8sx97\") pod \"maas-api-revoke-sub-46d37e92-phvgz\" (UID: \"def6461d-2480-45dc-b7cd-9f7047855ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d37e92-phvgz" Oct 29 23:14:58.761655 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.761619 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/def6461d-2480-45dc-b7cd-9f7047855ca7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46d37e92-phvgz\" (UID: \"def6461d-2480-45dc-b7cd-9f7047855ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d37e92-phvgz" Oct 29 23:14:58.761872 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.761856 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/def6461d-2480-45dc-b7cd-9f7047855ca7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46d37e92-phvgz\" (UID: \"def6461d-2480-45dc-b7cd-9f7047855ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d37e92-phvgz" Oct 29 23:14:58.863083 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.863052 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/def6461d-2480-45dc-b7cd-9f7047855ca7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46d37e92-phvgz\" (UID: \"def6461d-2480-45dc-b7cd-9f7047855ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d37e92-phvgz" Oct 29 23:14:58.863304 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.863151 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sx97\" (UniqueName: \"kubernetes.io/projected/def6461d-2480-45dc-b7cd-9f7047855ca7-kube-api-access-8sx97\") pod \"maas-api-revoke-sub-46d37e92-phvgz\" (UID: \"def6461d-2480-45dc-b7cd-9f7047855ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d37e92-phvgz" Oct 29 23:14:58.863304 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.863172 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/def6461d-2480-45dc-b7cd-9f7047855ca7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46d37e92-phvgz\" (UID: \"def6461d-2480-45dc-b7cd-9f7047855ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d37e92-phvgz" Oct 29 23:14:58.863807 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.863782 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/def6461d-2480-45dc-b7cd-9f7047855ca7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46d37e92-phvgz\" (UID: \"def6461d-2480-45dc-b7cd-9f7047855ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d37e92-phvgz" Oct 29 23:14:58.865772 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.865751 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/def6461d-2480-45dc-b7cd-9f7047855ca7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46d37e92-phvgz\" (UID: \"def6461d-2480-45dc-b7cd-9f7047855ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d37e92-phvgz" Oct 29 23:14:58.890777 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.890737 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sx97\" (UniqueName: \"kubernetes.io/projected/def6461d-2480-45dc-b7cd-9f7047855ca7-kube-api-access-8sx97\") pod \"maas-api-revoke-sub-46d37e92-phvgz\" (UID: \"def6461d-2480-45dc-b7cd-9f7047855ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d37e92-phvgz" Oct 29 23:14:58.986647 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:58.986614 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d37e92-phvgz" Oct 29 23:14:59.321988 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:59.321956 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46d37e92-phvgz"] Oct 29 23:14:59.322188 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:14:59.322166 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef6461d_2480_45dc_b7cd_9f7047855ca7.slice/crio-114de4b0ae285671c4282e9521175f947f4382979261d897cb59bf89fdd04f56 WatchSource:0}: Error finding container 114de4b0ae285671c4282e9521175f947f4382979261d897cb59bf89fdd04f56: Status 404 returned error can't find the container with id 114de4b0ae285671c4282e9521175f947f4382979261d897cb59bf89fdd04f56 Oct 29 23:14:59.534943 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:59.534910 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d37e92-phvgz" event={"ID":"def6461d-2480-45dc-b7cd-9f7047855ca7","Type":"ContainerStarted","Data":"114de4b0ae285671c4282e9521175f947f4382979261d897cb59bf89fdd04f56"} Oct 29 23:14:59.535365 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:59.535005 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7605-61b098" podUID="c20ecb28-30a7-4c78-bfd4-8c0896baf52f" containerName="test-curl-7605-61b098" containerID="cri-o://7445d4f91338a62208c881ea6865d01c95e56dce6365c4dca7c0ef8e4c76d7e6" gracePeriod=2 Oct 29 23:14:59.550059 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:14:59.550030 4821 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7605-61b098\" is forbidden: User \"system:node:ip-10-0-130-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-252.ec2.internal' and this object" podUID="c20ecb28-30a7-4c78-bfd4-8c0896baf52f" pod="opendatahub/test-curl-7605-61b098" Oct 29 23:14:59.794007 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:59.793974 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs"] Oct 29 23:14:59.798481 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:59.798450 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" Oct 29 23:14:59.813411 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:14:59.813375 4821 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7605-61b098\" is forbidden: User \"system:node:ip-10-0-130-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-252.ec2.internal' and this object" podUID="c20ecb28-30a7-4c78-bfd4-8c0896baf52f" pod="opendatahub/test-curl-7605-61b098" Oct 29 23:14:59.824376 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:59.824343 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs"] Oct 29 23:14:59.864834 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:59.864802 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-59eefa-serving-cert\"" Oct 29 23:14:59.882434 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:59.882406 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40800b56-bbd2-4274-8031-58667613c72a-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs\" (UID: \"40800b56-bbd2-4274-8031-58667613c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" Oct 29 23:14:59.882639 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:59.882497 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjrn7\" (UniqueName: \"kubernetes.io/projected/40800b56-bbd2-4274-8031-58667613c72a-kube-api-access-qjrn7\") pod \"maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs\" (UID: \"40800b56-bbd2-4274-8031-58667613c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" Oct 29 23:14:59.882639 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:59.882594 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40800b56-bbd2-4274-8031-58667613c72a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs\" (UID: \"40800b56-bbd2-4274-8031-58667613c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" Oct 29 23:14:59.883008 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:59.882985 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40800b56-bbd2-4274-8031-58667613c72a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs\" (UID: \"40800b56-bbd2-4274-8031-58667613c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" Oct 29 23:14:59.984626 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:59.984543 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjrn7\" (UniqueName: \"kubernetes.io/projected/40800b56-bbd2-4274-8031-58667613c72a-kube-api-access-qjrn7\") pod \"maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs\" (UID: \"40800b56-bbd2-4274-8031-58667613c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" Oct 29 23:14:59.984626 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:59.984591 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40800b56-bbd2-4274-8031-58667613c72a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs\" (UID: \"40800b56-bbd2-4274-8031-58667613c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" Oct 29 23:14:59.984879 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:59.984825 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40800b56-bbd2-4274-8031-58667613c72a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs\" (UID: \"40800b56-bbd2-4274-8031-58667613c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" Oct 29 23:14:59.984879 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:59.984856 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40800b56-bbd2-4274-8031-58667613c72a-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs\" (UID: \"40800b56-bbd2-4274-8031-58667613c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" Oct 29 23:14:59.984978 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:14:59.984958 4821 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-serving-cert: secret "maas-api-e2e-shared-a-w1-59eefa-serving-cert" not found Oct 29 23:14:59.985049 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:14:59.985038 4821 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40800b56-bbd2-4274-8031-58667613c72a-maas-api-tls podName:40800b56-bbd2-4274-8031-58667613c72a nodeName:}" failed. No retries permitted until 2026-10-29 23:15:00.485014393 +0000 UTC m=+1954.425562300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/40800b56-bbd2-4274-8031-58667613c72a-maas-api-tls") pod "maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" (UID: "40800b56-bbd2-4274-8031-58667613c72a") : secret "maas-api-e2e-shared-a-w1-59eefa-serving-cert" not found Oct 29 23:14:59.985376 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:59.985354 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40800b56-bbd2-4274-8031-58667613c72a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs\" (UID: \"40800b56-bbd2-4274-8031-58667613c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" Oct 29 23:14:59.987609 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:14:59.987590 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40800b56-bbd2-4274-8031-58667613c72a-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs\" (UID: \"40800b56-bbd2-4274-8031-58667613c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" Oct 29 23:15:00.048064 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.048030 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjrn7\" (UniqueName: \"kubernetes.io/projected/40800b56-bbd2-4274-8031-58667613c72a-kube-api-access-qjrn7\") pod \"maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs\" (UID: \"40800b56-bbd2-4274-8031-58667613c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" Oct 29 23:15:00.150769 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.150736 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm"] Oct 29 23:15:00.155926 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.155900 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97"] Oct 29 23:15:00.156182 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.156026 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm" Oct 29 23:15:00.160813 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.160787 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97" Oct 29 23:15:00.175610 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:00.175568 4821 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7605-61b098\" is forbidden: User \"system:node:ip-10-0-130-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-252.ec2.internal' and this object" podUID="c20ecb28-30a7-4c78-bfd4-8c0896baf52f" pod="opendatahub/test-curl-7605-61b098" Oct 29 23:15:00.298956 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.298641 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vnjr\" (UniqueName: \"kubernetes.io/projected/2b421f85-4894-47f3-9e87-c5b683ea4e6b-kube-api-access-8vnjr\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm\" (UID: \"2b421f85-4894-47f3-9e87-c5b683ea4e6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm" Oct 29 23:15:00.298956 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.298713 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b421f85-4894-47f3-9e87-c5b683ea4e6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm\" (UID: \"2b421f85-4894-47f3-9e87-c5b683ea4e6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm" Oct 29 23:15:00.298956 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.298784 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/514b5faa-73ee-4d54-9e6b-c474a52d82ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97\" (UID: \"514b5faa-73ee-4d54-9e6b-c474a52d82ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97" Oct 29 23:15:00.298956 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.298842 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mkmc\" (UniqueName: \"kubernetes.io/projected/514b5faa-73ee-4d54-9e6b-c474a52d82ca-kube-api-access-6mkmc\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97\" (UID: \"514b5faa-73ee-4d54-9e6b-c474a52d82ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97" Oct 29 23:15:00.301450 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:00.301417 4821 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7605-61b098\" is forbidden: User \"system:node:ip-10-0-130-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-252.ec2.internal' and this object" podUID="c20ecb28-30a7-4c78-bfd4-8c0896baf52f" pod="opendatahub/test-curl-7605-61b098" Oct 29 23:15:00.307032 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.307002 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm"] Oct 29 23:15:00.307167 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.307082 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g"] Oct 29 23:15:00.316275 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.313150 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97"] Oct 29 23:15:00.316275 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.313267 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp"] Oct 29 23:15:00.316275 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.315309 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g" Oct 29 23:15:00.326259 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.324690 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf"] Oct 29 23:15:00.326259 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.324852 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp" Oct 29 23:15:00.330442 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.330399 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g"] Oct 29 23:15:00.330591 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.330462 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp"] Oct 29 23:15:00.330591 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.330507 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf"] Oct 29 23:15:00.330591 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.330533 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6"] Oct 29 23:15:00.330591 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.330564 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf" Oct 29 23:15:00.337879 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.337669 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8"] Oct 29 23:15:00.339390 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.339363 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6" Oct 29 23:15:00.343927 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.343905 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5"] Oct 29 23:15:00.344054 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.344011 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8" Oct 29 23:15:00.350969 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.350941 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6"] Oct 29 23:15:00.351189 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.351106 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5"] Oct 29 23:15:00.351300 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.351038 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5" Oct 29 23:15:00.361336 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.361307 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8"] Oct 29 23:15:00.400165 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.400123 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/514b5faa-73ee-4d54-9e6b-c474a52d82ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97\" (UID: \"514b5faa-73ee-4d54-9e6b-c474a52d82ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97" Oct 29 23:15:00.400425 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.400218 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mkmc\" (UniqueName: \"kubernetes.io/projected/514b5faa-73ee-4d54-9e6b-c474a52d82ca-kube-api-access-6mkmc\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97\" (UID: \"514b5faa-73ee-4d54-9e6b-c474a52d82ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97" Oct 29 23:15:00.400425 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.400255 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0219f30-2070-4cf8-bdfa-304ad5c86ad7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5\" (UID: \"c0219f30-2070-4cf8-bdfa-304ad5c86ad7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5" Oct 29 23:15:00.400425 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.400308 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e44a4a-8eb9-4732-8924-d76683d1e213-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g\" (UID: \"17e44a4a-8eb9-4732-8924-d76683d1e213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g" Oct 29 23:15:00.400618 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.400507 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f8v7\" (UniqueName: \"kubernetes.io/projected/c0219f30-2070-4cf8-bdfa-304ad5c86ad7-kube-api-access-8f8v7\") pod \"maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5\" (UID: \"c0219f30-2070-4cf8-bdfa-304ad5c86ad7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5" Oct 29 23:15:00.400618 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.400555 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwmkq\" (UniqueName: \"kubernetes.io/projected/7b4b604b-3757-4c03-a745-f01e02ce82c2-kube-api-access-jwmkq\") pod \"maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8\" (UID: \"7b4b604b-3757-4c03-a745-f01e02ce82c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8" Oct 29 23:15:00.401061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.400633 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff81e511-3cab-4008-b2a5-1ab9774eeff4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp\" (UID: \"ff81e511-3cab-4008-b2a5-1ab9774eeff4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp" Oct 29 23:15:00.401061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.400713 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bc6m\" (UniqueName: \"kubernetes.io/projected/786f75a2-8350-4081-8ca4-041ed3247252-kube-api-access-5bc6m\") pod \"maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf\" (UID: \"786f75a2-8350-4081-8ca4-041ed3247252\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf" Oct 29 23:15:00.401061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.400790 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk5hn\" (UniqueName: \"kubernetes.io/projected/51189101-afc0-42dd-815e-a8330ec8b42a-kube-api-access-fk5hn\") pod \"maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6\" (UID: \"51189101-afc0-42dd-815e-a8330ec8b42a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6" Oct 29 23:15:00.401061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.400902 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/514b5faa-73ee-4d54-9e6b-c474a52d82ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97\" (UID: \"514b5faa-73ee-4d54-9e6b-c474a52d82ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97" Oct 29 23:15:00.401061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.400916 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5jtr\" (UniqueName: \"kubernetes.io/projected/ff81e511-3cab-4008-b2a5-1ab9774eeff4-kube-api-access-h5jtr\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp\" (UID: \"ff81e511-3cab-4008-b2a5-1ab9774eeff4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp" Oct 29 23:15:00.401061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.401021 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b4b604b-3757-4c03-a745-f01e02ce82c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8\" (UID: \"7b4b604b-3757-4c03-a745-f01e02ce82c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8" Oct 29 23:15:00.401410 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.401078 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786f75a2-8350-4081-8ca4-041ed3247252-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf\" (UID: \"786f75a2-8350-4081-8ca4-041ed3247252\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf" Oct 29 23:15:00.401410 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.401114 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vnjr\" (UniqueName: \"kubernetes.io/projected/2b421f85-4894-47f3-9e87-c5b683ea4e6b-kube-api-access-8vnjr\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm\" (UID: \"2b421f85-4894-47f3-9e87-c5b683ea4e6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm" Oct 29 23:15:00.401410 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.401331 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b421f85-4894-47f3-9e87-c5b683ea4e6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm\" (UID: \"2b421f85-4894-47f3-9e87-c5b683ea4e6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm" Oct 29 23:15:00.401410 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.401371 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pskb6\" (UniqueName: \"kubernetes.io/projected/17e44a4a-8eb9-4732-8924-d76683d1e213-kube-api-access-pskb6\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g\" (UID: \"17e44a4a-8eb9-4732-8924-d76683d1e213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g" Oct 29 23:15:00.401573 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.401417 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51189101-afc0-42dd-815e-a8330ec8b42a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6\" (UID: \"51189101-afc0-42dd-815e-a8330ec8b42a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6" Oct 29 23:15:00.402071 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.402051 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b421f85-4894-47f3-9e87-c5b683ea4e6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm\" (UID: \"2b421f85-4894-47f3-9e87-c5b683ea4e6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm" Oct 29 23:15:00.425532 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.425510 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vnjr\" (UniqueName: \"kubernetes.io/projected/2b421f85-4894-47f3-9e87-c5b683ea4e6b-kube-api-access-8vnjr\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm\" (UID: \"2b421f85-4894-47f3-9e87-c5b683ea4e6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm" Oct 29 23:15:00.425906 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.425878 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mkmc\" (UniqueName: \"kubernetes.io/projected/514b5faa-73ee-4d54-9e6b-c474a52d82ca-kube-api-access-6mkmc\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97\" (UID: \"514b5faa-73ee-4d54-9e6b-c474a52d82ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97" Oct 29 23:15:00.502577 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.502544 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0219f30-2070-4cf8-bdfa-304ad5c86ad7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5\" (UID: \"c0219f30-2070-4cf8-bdfa-304ad5c86ad7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5" Oct 29 23:15:00.502764 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.502601 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e44a4a-8eb9-4732-8924-d76683d1e213-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g\" (UID: \"17e44a4a-8eb9-4732-8924-d76683d1e213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g" Oct 29 23:15:00.502764 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.502651 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40800b56-bbd2-4274-8031-58667613c72a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs\" (UID: \"40800b56-bbd2-4274-8031-58667613c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" Oct 29 23:15:00.502764 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.502688 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f8v7\" (UniqueName: \"kubernetes.io/projected/c0219f30-2070-4cf8-bdfa-304ad5c86ad7-kube-api-access-8f8v7\") pod \"maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5\" (UID: \"c0219f30-2070-4cf8-bdfa-304ad5c86ad7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5" Oct 29 23:15:00.502764 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.502722 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwmkq\" (UniqueName: \"kubernetes.io/projected/7b4b604b-3757-4c03-a745-f01e02ce82c2-kube-api-access-jwmkq\") pod \"maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8\" (UID: \"7b4b604b-3757-4c03-a745-f01e02ce82c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8" Oct 29 23:15:00.502764 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.502756 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff81e511-3cab-4008-b2a5-1ab9774eeff4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp\" (UID: \"ff81e511-3cab-4008-b2a5-1ab9774eeff4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp" Oct 29 23:15:00.503071 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.502796 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bc6m\" (UniqueName: \"kubernetes.io/projected/786f75a2-8350-4081-8ca4-041ed3247252-kube-api-access-5bc6m\") pod \"maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf\" (UID: \"786f75a2-8350-4081-8ca4-041ed3247252\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf" Oct 29 23:15:00.503071 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.502859 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk5hn\" (UniqueName: \"kubernetes.io/projected/51189101-afc0-42dd-815e-a8330ec8b42a-kube-api-access-fk5hn\") pod \"maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6\" (UID: \"51189101-afc0-42dd-815e-a8330ec8b42a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6" Oct 29 23:15:00.503071 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.502965 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5jtr\" (UniqueName: \"kubernetes.io/projected/ff81e511-3cab-4008-b2a5-1ab9774eeff4-kube-api-access-h5jtr\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp\" (UID: \"ff81e511-3cab-4008-b2a5-1ab9774eeff4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp" Oct 29 23:15:00.503071 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.503036 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b4b604b-3757-4c03-a745-f01e02ce82c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8\" (UID: \"7b4b604b-3757-4c03-a745-f01e02ce82c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8" Oct 29 23:15:00.503396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.503097 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786f75a2-8350-4081-8ca4-041ed3247252-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf\" (UID: \"786f75a2-8350-4081-8ca4-041ed3247252\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf" Oct 29 23:15:00.503396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.503182 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pskb6\" (UniqueName: \"kubernetes.io/projected/17e44a4a-8eb9-4732-8924-d76683d1e213-kube-api-access-pskb6\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g\" (UID: \"17e44a4a-8eb9-4732-8924-d76683d1e213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g" Oct 29 23:15:00.503396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.503238 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51189101-afc0-42dd-815e-a8330ec8b42a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6\" (UID: \"51189101-afc0-42dd-815e-a8330ec8b42a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6" Oct 29 23:15:00.503591 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.503391 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0219f30-2070-4cf8-bdfa-304ad5c86ad7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5\" (UID: \"c0219f30-2070-4cf8-bdfa-304ad5c86ad7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5" Oct 29 23:15:00.503591 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.503524 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e44a4a-8eb9-4732-8924-d76683d1e213-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g\" (UID: \"17e44a4a-8eb9-4732-8924-d76683d1e213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g" Oct 29 23:15:00.503876 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.503855 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff81e511-3cab-4008-b2a5-1ab9774eeff4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp\" (UID: \"ff81e511-3cab-4008-b2a5-1ab9774eeff4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp" Oct 29 23:15:00.503934 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.503855 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b4b604b-3757-4c03-a745-f01e02ce82c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8\" (UID: \"7b4b604b-3757-4c03-a745-f01e02ce82c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8" Oct 29 23:15:00.503971 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.503961 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51189101-afc0-42dd-815e-a8330ec8b42a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6\" (UID: \"51189101-afc0-42dd-815e-a8330ec8b42a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6" Oct 29 23:15:00.504184 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.504160 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786f75a2-8350-4081-8ca4-041ed3247252-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf\" (UID: \"786f75a2-8350-4081-8ca4-041ed3247252\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf" Oct 29 23:15:00.505519 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.505502 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40800b56-bbd2-4274-8031-58667613c72a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs\" (UID: \"40800b56-bbd2-4274-8031-58667613c72a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" Oct 29 23:15:00.508384 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.508365 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm" Oct 29 23:15:00.524977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.524957 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97" Oct 29 23:15:00.530371 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.530322 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5jtr\" (UniqueName: \"kubernetes.io/projected/ff81e511-3cab-4008-b2a5-1ab9774eeff4-kube-api-access-h5jtr\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp\" (UID: \"ff81e511-3cab-4008-b2a5-1ab9774eeff4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp" Oct 29 23:15:00.530498 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.530472 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwmkq\" (UniqueName: \"kubernetes.io/projected/7b4b604b-3757-4c03-a745-f01e02ce82c2-kube-api-access-jwmkq\") pod \"maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8\" (UID: \"7b4b604b-3757-4c03-a745-f01e02ce82c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8" Oct 29 23:15:00.530575 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.530558 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk5hn\" (UniqueName: \"kubernetes.io/projected/51189101-afc0-42dd-815e-a8330ec8b42a-kube-api-access-fk5hn\") pod \"maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6\" (UID: \"51189101-afc0-42dd-815e-a8330ec8b42a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6" Oct 29 23:15:00.530945 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.530923 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f8v7\" (UniqueName: \"kubernetes.io/projected/c0219f30-2070-4cf8-bdfa-304ad5c86ad7-kube-api-access-8f8v7\") pod \"maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5\" (UID: \"c0219f30-2070-4cf8-bdfa-304ad5c86ad7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5" Oct 29 23:15:00.531006 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.530923 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bc6m\" (UniqueName: \"kubernetes.io/projected/786f75a2-8350-4081-8ca4-041ed3247252-kube-api-access-5bc6m\") pod \"maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf\" (UID: \"786f75a2-8350-4081-8ca4-041ed3247252\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf" Oct 29 23:15:00.531006 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.530995 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pskb6\" (UniqueName: \"kubernetes.io/projected/17e44a4a-8eb9-4732-8924-d76683d1e213-kube-api-access-pskb6\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g\" (UID: \"17e44a4a-8eb9-4732-8924-d76683d1e213\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g" Oct 29 23:15:00.548677 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.548650 4821 generic.go:356] "Generic (PLEG): container finished" podID="def6461d-2480-45dc-b7cd-9f7047855ca7" containerID="0d2f4f9130a6858a01ecdb0438ec1af39f7e003b07fdf40f644832e4aec04d2d" exitCode=0 Oct 29 23:15:00.549009 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.548687 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d37e92-phvgz" event={"ID":"def6461d-2480-45dc-b7cd-9f7047855ca7","Type":"ContainerDied","Data":"0d2f4f9130a6858a01ecdb0438ec1af39f7e003b07fdf40f644832e4aec04d2d"} Oct 29 23:15:00.709094 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.709052 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g" Oct 29 23:15:00.712855 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.712828 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" Oct 29 23:15:00.717009 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.716835 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp" Oct 29 23:15:00.726731 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.725612 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf" Oct 29 23:15:00.753567 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.753509 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6" Oct 29 23:15:00.760622 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.760472 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8" Oct 29 23:15:00.767379 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.767178 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5" Oct 29 23:15:00.884232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.882043 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm"] Oct 29 23:15:00.888831 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:00.888780 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97"] Oct 29 23:15:00.964171 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:15:00.964139 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b421f85_4894_47f3_9e87_c5b683ea4e6b.slice/crio-67fa5e42d1e1865e6ed36d37ddeb10b3d7b8d214cb7da8c3ea43d2e6829575e8 WatchSource:0}: Error finding container 67fa5e42d1e1865e6ed36d37ddeb10b3d7b8d214cb7da8c3ea43d2e6829575e8: Status 404 returned error can't find the container with id 67fa5e42d1e1865e6ed36d37ddeb10b3d7b8d214cb7da8c3ea43d2e6829575e8 Oct 29 23:15:01.143798 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:01.142520 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g"] Oct 29 23:15:01.700052 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:01.678181 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp"] Oct 29 23:15:01.794024 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:01.793910 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm" event={"ID":"2b421f85-4894-47f3-9e87-c5b683ea4e6b","Type":"ContainerStarted","Data":"67fa5e42d1e1865e6ed36d37ddeb10b3d7b8d214cb7da8c3ea43d2e6829575e8"} Oct 29 23:15:01.809927 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:01.795690 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf"] Oct 29 23:15:01.815324 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:01.815289 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97" event={"ID":"514b5faa-73ee-4d54-9e6b-c474a52d82ca","Type":"ContainerStarted","Data":"5482b8e512f66652e09a544c6b807b6c7ccb6517153db9594916bd57b65aa12d"} Oct 29 23:15:01.830251 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:01.829281 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g" event={"ID":"17e44a4a-8eb9-4732-8924-d76683d1e213","Type":"ContainerStarted","Data":"f3bb72277fb9d0aca2fdf67f2402733e98aa59158bab897cc4af6ee1a3a76824"} Oct 29 23:15:02.023427 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.022629 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5"] Oct 29 23:15:02.046304 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.042278 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs"] Oct 29 23:15:02.092964 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.092855 4821 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:15:02.187179 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.187151 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7605-61b098" Oct 29 23:15:02.216149 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.216114 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8"] Oct 29 23:15:02.220952 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.220689 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6"] Oct 29 23:15:02.239186 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.239161 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d37e92-phvgz" Oct 29 23:15:02.261599 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.261566 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/def6461d-2480-45dc-b7cd-9f7047855ca7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/def6461d-2480-45dc-b7cd-9f7047855ca7-maas-api-openshift-ca\") pod \"def6461d-2480-45dc-b7cd-9f7047855ca7\" (UID: \"def6461d-2480-45dc-b7cd-9f7047855ca7\") " Oct 29 23:15:02.261760 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.261665 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/def6461d-2480-45dc-b7cd-9f7047855ca7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/def6461d-2480-45dc-b7cd-9f7047855ca7-maas-api-xks-ca\") pod \"def6461d-2480-45dc-b7cd-9f7047855ca7\" (UID: \"def6461d-2480-45dc-b7cd-9f7047855ca7\") " Oct 29 23:15:02.261760 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.261719 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def6461d-2480-45dc-b7cd-9f7047855ca7-kube-api-access-8sx97\" (UniqueName: \"kubernetes.io/projected/def6461d-2480-45dc-b7cd-9f7047855ca7-kube-api-access-8sx97\") pod \"def6461d-2480-45dc-b7cd-9f7047855ca7\" (UID: \"def6461d-2480-45dc-b7cd-9f7047855ca7\") " Oct 29 23:15:02.261977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.261954 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def6461d-2480-45dc-b7cd-9f7047855ca7-maas-api-openshift-ca" pod "def6461d-2480-45dc-b7cd-9f7047855ca7" (UID: "def6461d-2480-45dc-b7cd-9f7047855ca7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:02.262045 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.261995 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c20ecb28-30a7-4c78-bfd4-8c0896baf52f-kube-api-access-d8xsz\" (UniqueName: \"kubernetes.io/projected/c20ecb28-30a7-4c78-bfd4-8c0896baf52f-kube-api-access-d8xsz\") pod \"c20ecb28-30a7-4c78-bfd4-8c0896baf52f\" (UID: \"c20ecb28-30a7-4c78-bfd4-8c0896baf52f\") " Oct 29 23:15:02.263286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.263257 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/def6461d-2480-45dc-b7cd-9f7047855ca7-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:02.421039 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.420104 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def6461d-2480-45dc-b7cd-9f7047855ca7-kube-api-access-8sx97" pod "def6461d-2480-45dc-b7cd-9f7047855ca7" (UID: "def6461d-2480-45dc-b7cd-9f7047855ca7"). InnerVolumeSpecName "kube-api-access-8sx97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:02.423959 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.423910 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c20ecb28-30a7-4c78-bfd4-8c0896baf52f-kube-api-access-d8xsz" pod "c20ecb28-30a7-4c78-bfd4-8c0896baf52f" (UID: "c20ecb28-30a7-4c78-bfd4-8c0896baf52f"). InnerVolumeSpecName "kube-api-access-d8xsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:02.424239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.423966 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def6461d-2480-45dc-b7cd-9f7047855ca7-maas-api-xks-ca" pod "def6461d-2480-45dc-b7cd-9f7047855ca7" (UID: "def6461d-2480-45dc-b7cd-9f7047855ca7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:15:02.466657 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.466626 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/def6461d-2480-45dc-b7cd-9f7047855ca7-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:02.466657 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.466660 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sx97\" (UniqueName: \"kubernetes.io/projected/def6461d-2480-45dc-b7cd-9f7047855ca7-kube-api-access-8sx97\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:02.467185 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.466674 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8xsz\" (UniqueName: \"kubernetes.io/projected/c20ecb28-30a7-4c78-bfd4-8c0896baf52f-kube-api-access-d8xsz\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:02.843394 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.843357 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" event={"ID":"40800b56-bbd2-4274-8031-58667613c72a","Type":"ContainerStarted","Data":"0434ed809cc903182428208fe73a95df3f46a336189716c4992ff72a73c01f8a"} Oct 29 23:15:02.850677 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.850193 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d37e92-phvgz" event={"ID":"def6461d-2480-45dc-b7cd-9f7047855ca7","Type":"ContainerDied","Data":"114de4b0ae285671c4282e9521175f947f4382979261d897cb59bf89fdd04f56"} Oct 29 23:15:02.850677 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.850295 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="114de4b0ae285671c4282e9521175f947f4382979261d897cb59bf89fdd04f56" Oct 29 23:15:02.850677 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.850375 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46d37e92-phvgz" Oct 29 23:15:02.856026 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.855944 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf" event={"ID":"786f75a2-8350-4081-8ca4-041ed3247252","Type":"ContainerStarted","Data":"9ad67eca7c475ad11ff85c78dc2a9a16431d9a099d3427d7dbf039e6bf3b4ada"} Oct 29 23:15:02.856026 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.855996 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf" event={"ID":"786f75a2-8350-4081-8ca4-041ed3247252","Type":"ContainerStarted","Data":"15ddd722c083028be0854a0e13ebc6c388d0584cabbbb8ea25485fdb9d1bdb4c"} Oct 29 23:15:02.858898 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.858874 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6" event={"ID":"51189101-afc0-42dd-815e-a8330ec8b42a","Type":"ContainerStarted","Data":"1eaa2234227e7659903c647ed63048cb30d0d0d9840e4915b4f83638387e68fd"} Oct 29 23:15:02.860968 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.860830 4821 generic.go:356] "Generic (PLEG): container finished" podID="17e44a4a-8eb9-4732-8924-d76683d1e213" containerID="4ae4e6d2d503faf4555bde0cdd26dac7838c54301445117e3b57fc5d408dd2fc" exitCode=60 Oct 29 23:15:02.860968 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.860938 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g" event={"ID":"17e44a4a-8eb9-4732-8924-d76683d1e213","Type":"ContainerDied","Data":"4ae4e6d2d503faf4555bde0cdd26dac7838c54301445117e3b57fc5d408dd2fc"} Oct 29 23:15:02.861476 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.861456 4821 scope.go:122] "RemoveContainer" containerID="4ae4e6d2d503faf4555bde0cdd26dac7838c54301445117e3b57fc5d408dd2fc" Oct 29 23:15:02.864861 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.864813 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8" event={"ID":"7b4b604b-3757-4c03-a745-f01e02ce82c2","Type":"ContainerStarted","Data":"43f8d273d19c2e7a29e6563ea0407cabaae6f03abb0fd03e28b54e742d50dbcb"} Oct 29 23:15:02.866851 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.866592 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp" event={"ID":"ff81e511-3cab-4008-b2a5-1ab9774eeff4","Type":"ContainerStarted","Data":"7ca9927aab2aef219fa8531fe6a3bb02f56639fa0047ef6a7645081680673256"} Oct 29 23:15:02.866851 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.866625 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp" event={"ID":"ff81e511-3cab-4008-b2a5-1ab9774eeff4","Type":"ContainerStarted","Data":"3000b6901097762b061cb02a759c022076211042a1cabd99ea0a4811f6f26c38"} Oct 29 23:15:02.870082 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.870057 4821 generic.go:356] "Generic (PLEG): container finished" podID="2b421f85-4894-47f3-9e87-c5b683ea4e6b" containerID="c797d38ee15f1983d2275ab0a0b4424a77d3b84d3ae2e9407044159ce126ea54" exitCode=60 Oct 29 23:15:02.870397 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.870253 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm" event={"ID":"2b421f85-4894-47f3-9e87-c5b683ea4e6b","Type":"ContainerDied","Data":"c797d38ee15f1983d2275ab0a0b4424a77d3b84d3ae2e9407044159ce126ea54"} Oct 29 23:15:02.870622 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.870599 4821 scope.go:122] "RemoveContainer" containerID="c797d38ee15f1983d2275ab0a0b4424a77d3b84d3ae2e9407044159ce126ea54" Oct 29 23:15:02.874816 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.874784 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5" event={"ID":"c0219f30-2070-4cf8-bdfa-304ad5c86ad7","Type":"ContainerStarted","Data":"88104b40bc3ef550fd0bba169002993b97a4804ea0891ec8ccbe921911671a00"} Oct 29 23:15:02.874929 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.874857 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5" event={"ID":"c0219f30-2070-4cf8-bdfa-304ad5c86ad7","Type":"ContainerStarted","Data":"e3bbb6509c0b73acf60504bfb9404a63854d4b24a5a6a030eec22f71b9011631"} Oct 29 23:15:02.881219 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.876838 4821 generic.go:356] "Generic (PLEG): container finished" podID="c20ecb28-30a7-4c78-bfd4-8c0896baf52f" containerID="7445d4f91338a62208c881ea6865d01c95e56dce6365c4dca7c0ef8e4c76d7e6" exitCode=137 Oct 29 23:15:02.881219 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.876915 4821 scope.go:122] "RemoveContainer" containerID="7445d4f91338a62208c881ea6865d01c95e56dce6365c4dca7c0ef8e4c76d7e6" Oct 29 23:15:02.881219 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.877076 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7605-61b098" Oct 29 23:15:02.893879 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.893850 4821 generic.go:356] "Generic (PLEG): container finished" podID="514b5faa-73ee-4d54-9e6b-c474a52d82ca" containerID="40e7d7e2fcefc302558f9face78f1e2b0d0d236d77de098b1a43c3618307f90c" exitCode=60 Oct 29 23:15:02.893987 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.893917 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97" event={"ID":"514b5faa-73ee-4d54-9e6b-c474a52d82ca","Type":"ContainerDied","Data":"40e7d7e2fcefc302558f9face78f1e2b0d0d236d77de098b1a43c3618307f90c"} Oct 29 23:15:02.894383 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.894359 4821 scope.go:122] "RemoveContainer" containerID="40e7d7e2fcefc302558f9face78f1e2b0d0d236d77de098b1a43c3618307f90c" Oct 29 23:15:02.906296 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.906250 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf" podStartSLOduration=2.906233415 podStartE2EDuration="2.906233415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:02.903677453 +0000 UTC m=+1956.844225363" watchObservedRunningTime="2026-10-29 23:15:02.906233415 +0000 UTC m=+1956.846781325" Oct 29 23:15:02.959116 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.959090 4821 scope.go:122] "RemoveContainer" containerID="7445d4f91338a62208c881ea6865d01c95e56dce6365c4dca7c0ef8e4c76d7e6" Oct 29 23:15:02.959500 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:02.959471 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7445d4f91338a62208c881ea6865d01c95e56dce6365c4dca7c0ef8e4c76d7e6\": container with ID starting with 7445d4f91338a62208c881ea6865d01c95e56dce6365c4dca7c0ef8e4c76d7e6 not found: ID does not exist" containerID="7445d4f91338a62208c881ea6865d01c95e56dce6365c4dca7c0ef8e4c76d7e6" Oct 29 23:15:02.959637 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.959514 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7445d4f91338a62208c881ea6865d01c95e56dce6365c4dca7c0ef8e4c76d7e6"} err="failed to get container status \"7445d4f91338a62208c881ea6865d01c95e56dce6365c4dca7c0ef8e4c76d7e6\": rpc error: code = NotFound desc = could not find container \"7445d4f91338a62208c881ea6865d01c95e56dce6365c4dca7c0ef8e4c76d7e6\": container with ID starting with 7445d4f91338a62208c881ea6865d01c95e56dce6365c4dca7c0ef8e4c76d7e6 not found: ID does not exist" Oct 29 23:15:02.972396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:02.972330 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp" podStartSLOduration=2.9723146099999997 podStartE2EDuration="2.97231461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:02.966689436 +0000 UTC m=+1956.907237344" watchObservedRunningTime="2026-10-29 23:15:02.97231461 +0000 UTC m=+1956.912862521" Oct 29 23:15:03.099706 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:03.099641 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5" podStartSLOduration=3.099621753 podStartE2EDuration="3.099621753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:03.098136701 +0000 UTC m=+1957.038684612" watchObservedRunningTime="2026-10-29 23:15:03.099621753 +0000 UTC m=+1957.040169668" Oct 29 23:15:03.904374 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:03.904327 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" event={"ID":"40800b56-bbd2-4274-8031-58667613c72a","Type":"ContainerStarted","Data":"f138c4d8fdeacf08c714ac1e8db914883556ffabd154378a7b51a4dfa5929070"} Oct 29 23:15:03.904828 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:03.904399 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" Oct 29 23:15:03.906175 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:03.906136 4821 generic.go:356] "Generic (PLEG): container finished" podID="786f75a2-8350-4081-8ca4-041ed3247252" containerID="9ad67eca7c475ad11ff85c78dc2a9a16431d9a099d3427d7dbf039e6bf3b4ada" exitCode=60 Oct 29 23:15:03.906332 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:03.906194 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf" event={"ID":"786f75a2-8350-4081-8ca4-041ed3247252","Type":"ContainerDied","Data":"9ad67eca7c475ad11ff85c78dc2a9a16431d9a099d3427d7dbf039e6bf3b4ada"} Oct 29 23:15:03.906517 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:03.906503 4821 scope.go:122] "RemoveContainer" containerID="9ad67eca7c475ad11ff85c78dc2a9a16431d9a099d3427d7dbf039e6bf3b4ada" Oct 29 23:15:03.908054 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:03.908034 4821 generic.go:356] "Generic (PLEG): container finished" podID="51189101-afc0-42dd-815e-a8330ec8b42a" containerID="f8fc18fe03a8d0d5c9dc4009a65f7dd5cf39ec03c8de4faf1bf38d63ec80cba3" exitCode=60 Oct 29 23:15:03.908176 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:03.908117 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6" event={"ID":"51189101-afc0-42dd-815e-a8330ec8b42a","Type":"ContainerDied","Data":"f8fc18fe03a8d0d5c9dc4009a65f7dd5cf39ec03c8de4faf1bf38d63ec80cba3"} Oct 29 23:15:03.908397 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:03.908359 4821 scope.go:122] "RemoveContainer" containerID="f8fc18fe03a8d0d5c9dc4009a65f7dd5cf39ec03c8de4faf1bf38d63ec80cba3" Oct 29 23:15:03.910409 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:03.910388 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g" event={"ID":"17e44a4a-8eb9-4732-8924-d76683d1e213","Type":"ContainerStarted","Data":"56bba4deb2101d4c06fdf4fbc5f418a3629f63c23c9c84bc0128326f006bb2ba"} Oct 29 23:15:03.912140 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:03.912118 4821 generic.go:356] "Generic (PLEG): container finished" podID="7b4b604b-3757-4c03-a745-f01e02ce82c2" containerID="73e258409b5d6d0525ecefcce6051d9569e8a635ebb983c75466883edff5f16e" exitCode=60 Oct 29 23:15:03.912259 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:03.912192 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8" event={"ID":"7b4b604b-3757-4c03-a745-f01e02ce82c2","Type":"ContainerDied","Data":"73e258409b5d6d0525ecefcce6051d9569e8a635ebb983c75466883edff5f16e"} Oct 29 23:15:03.912456 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:03.912442 4821 scope.go:122] "RemoveContainer" containerID="73e258409b5d6d0525ecefcce6051d9569e8a635ebb983c75466883edff5f16e" Oct 29 23:15:03.913906 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:03.913884 4821 generic.go:356] "Generic (PLEG): container finished" podID="ff81e511-3cab-4008-b2a5-1ab9774eeff4" containerID="7ca9927aab2aef219fa8531fe6a3bb02f56639fa0047ef6a7645081680673256" exitCode=60 Oct 29 23:15:03.913993 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:03.913974 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp" event={"ID":"ff81e511-3cab-4008-b2a5-1ab9774eeff4","Type":"ContainerDied","Data":"7ca9927aab2aef219fa8531fe6a3bb02f56639fa0047ef6a7645081680673256"} Oct 29 23:15:03.914264 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:03.914190 4821 scope.go:122] "RemoveContainer" containerID="7ca9927aab2aef219fa8531fe6a3bb02f56639fa0047ef6a7645081680673256" Oct 29 23:15:03.916157 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:03.916131 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm" event={"ID":"2b421f85-4894-47f3-9e87-c5b683ea4e6b","Type":"ContainerStarted","Data":"7e80865fe582798b8c8aa6eb4018dc99cd1072f90baa43dae6723d5c1756437f"} Oct 29 23:15:03.917857 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:03.917834 4821 generic.go:356] "Generic (PLEG): container finished" podID="c0219f30-2070-4cf8-bdfa-304ad5c86ad7" containerID="88104b40bc3ef550fd0bba169002993b97a4804ea0891ec8ccbe921911671a00" exitCode=60 Oct 29 23:15:03.917932 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:03.917914 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5" event={"ID":"c0219f30-2070-4cf8-bdfa-304ad5c86ad7","Type":"ContainerDied","Data":"88104b40bc3ef550fd0bba169002993b97a4804ea0891ec8ccbe921911671a00"} Oct 29 23:15:03.918143 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:03.918125 4821 scope.go:122] "RemoveContainer" containerID="88104b40bc3ef550fd0bba169002993b97a4804ea0891ec8ccbe921911671a00" Oct 29 23:15:03.921307 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:03.921286 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97" event={"ID":"514b5faa-73ee-4d54-9e6b-c474a52d82ca","Type":"ContainerStarted","Data":"7e8343f852d45f97f8a0d9aa7fe7995388e2c76039af4a6d3cd8c67b55389dd6"} Oct 29 23:15:03.952423 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:03.952370 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" podStartSLOduration=4.298042571 podStartE2EDuration="4.952346398s" totalImagesPullingTime="654.303827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:03.949881515 +0000 UTC m=+1957.890429426" watchObservedRunningTime="2026-10-29 23:15:03.952346398 +0000 UTC m=+1957.892894309" Oct 29 23:15:03.996495 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:03.996426 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm" podStartSLOduration=3.9964132980000002 podStartE2EDuration="3.996413298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:03.992851249 +0000 UTC m=+1957.933399163" watchObservedRunningTime="2026-10-29 23:15:03.996413298 +0000 UTC m=+1957.936961207" Oct 29 23:15:04.045931 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:04.045864 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g" podStartSLOduration=4.045845162 podStartE2EDuration="4.045845162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:04.038282417 +0000 UTC m=+1957.978830328" watchObservedRunningTime="2026-10-29 23:15:04.045845162 +0000 UTC m=+1957.986393073" Oct 29 23:15:04.149006 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:04.147280 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97" podStartSLOduration=4.14725852 podStartE2EDuration="4.14725852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:04.137093214 +0000 UTC m=+1958.077641126" watchObservedRunningTime="2026-10-29 23:15:04.14725852 +0000 UTC m=+1958.087806430" Oct 29 23:15:04.173738 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:04.173707 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c20ecb28-30a7-4c78-bfd4-8c0896baf52f" path="/var/lib/kubelet/pods/c20ecb28-30a7-4c78-bfd4-8c0896baf52f/volumes" Oct 29 23:15:04.934418 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:04.934382 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf" event={"ID":"786f75a2-8350-4081-8ca4-041ed3247252","Type":"ContainerStarted","Data":"d676204ec9fe9ff1fd07988c60a6e267ade773c07960db5cfd54ffd03ac434a1"} Oct 29 23:15:04.936569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:04.936540 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6" event={"ID":"51189101-afc0-42dd-815e-a8330ec8b42a","Type":"ContainerStarted","Data":"2f0debc3f5ee1ba288366661eacb8fc64dd9144d241c53ffb83527116beea585"} Oct 29 23:15:04.938623 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:04.938598 4821 generic.go:356] "Generic (PLEG): container finished" podID="17e44a4a-8eb9-4732-8924-d76683d1e213" containerID="56bba4deb2101d4c06fdf4fbc5f418a3629f63c23c9c84bc0128326f006bb2ba" exitCode=60 Oct 29 23:15:04.938734 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:04.938672 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g" event={"ID":"17e44a4a-8eb9-4732-8924-d76683d1e213","Type":"ContainerDied","Data":"56bba4deb2101d4c06fdf4fbc5f418a3629f63c23c9c84bc0128326f006bb2ba"} Oct 29 23:15:04.938734 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:04.938721 4821 scope.go:122] "RemoveContainer" containerID="4ae4e6d2d503faf4555bde0cdd26dac7838c54301445117e3b57fc5d408dd2fc" Oct 29 23:15:04.938908 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:04.938884 4821 scope.go:122] "RemoveContainer" containerID="56bba4deb2101d4c06fdf4fbc5f418a3629f63c23c9c84bc0128326f006bb2ba" Oct 29 23:15:04.939262 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:04.939237 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g_odh-ai-gateway-infra(17e44a4a-8eb9-4732-8924-d76683d1e213)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g" podUID="17e44a4a-8eb9-4732-8924-d76683d1e213" Oct 29 23:15:04.940931 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:04.940911 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8" event={"ID":"7b4b604b-3757-4c03-a745-f01e02ce82c2","Type":"ContainerStarted","Data":"a6e9a8ceb935a0091a00ba9ab03fa67799bc2509bfb0a3038ec59a6cc908580f"} Oct 29 23:15:04.942890 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:04.942863 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp" event={"ID":"ff81e511-3cab-4008-b2a5-1ab9774eeff4","Type":"ContainerStarted","Data":"0cf93952be4d4f8c8abc86539a5656751215c9d94f0ffef80af02fdc63b6f86a"} Oct 29 23:15:04.944835 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:04.944813 4821 generic.go:356] "Generic (PLEG): container finished" podID="2b421f85-4894-47f3-9e87-c5b683ea4e6b" containerID="7e80865fe582798b8c8aa6eb4018dc99cd1072f90baa43dae6723d5c1756437f" exitCode=60 Oct 29 23:15:04.944931 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:04.944870 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm" event={"ID":"2b421f85-4894-47f3-9e87-c5b683ea4e6b","Type":"ContainerDied","Data":"7e80865fe582798b8c8aa6eb4018dc99cd1072f90baa43dae6723d5c1756437f"} Oct 29 23:15:04.945155 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:04.945134 4821 scope.go:122] "RemoveContainer" containerID="7e80865fe582798b8c8aa6eb4018dc99cd1072f90baa43dae6723d5c1756437f" Oct 29 23:15:04.945538 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:04.945510 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm_odh-ai-gateway-infra(2b421f85-4894-47f3-9e87-c5b683ea4e6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm" podUID="2b421f85-4894-47f3-9e87-c5b683ea4e6b" Oct 29 23:15:04.946983 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:04.946958 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5" event={"ID":"c0219f30-2070-4cf8-bdfa-304ad5c86ad7","Type":"ContainerStarted","Data":"2068bd6ab12c5743bfcbee87a3bd07029f7abc733a738e0d2becfa9023e28f73"} Oct 29 23:15:04.949128 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:04.949108 4821 generic.go:356] "Generic (PLEG): container finished" podID="514b5faa-73ee-4d54-9e6b-c474a52d82ca" containerID="7e8343f852d45f97f8a0d9aa7fe7995388e2c76039af4a6d3cd8c67b55389dd6" exitCode=60 Oct 29 23:15:04.949221 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:04.949187 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97" event={"ID":"514b5faa-73ee-4d54-9e6b-c474a52d82ca","Type":"ContainerDied","Data":"7e8343f852d45f97f8a0d9aa7fe7995388e2c76039af4a6d3cd8c67b55389dd6"} Oct 29 23:15:04.949416 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:04.949401 4821 scope.go:122] "RemoveContainer" containerID="7e8343f852d45f97f8a0d9aa7fe7995388e2c76039af4a6d3cd8c67b55389dd6" Oct 29 23:15:04.949662 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:04.949645 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97_odh-ai-gateway-infra(514b5faa-73ee-4d54-9e6b-c474a52d82ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97" podUID="514b5faa-73ee-4d54-9e6b-c474a52d82ca" Oct 29 23:15:04.954640 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:04.954623 4821 scope.go:122] "RemoveContainer" containerID="c797d38ee15f1983d2275ab0a0b4424a77d3b84d3ae2e9407044159ce126ea54" Oct 29 23:15:04.972714 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:04.972314 4821 scope.go:122] "RemoveContainer" containerID="40e7d7e2fcefc302558f9face78f1e2b0d0d236d77de098b1a43c3618307f90c" Oct 29 23:15:05.211494 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:05.211430 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6" podStartSLOduration=5.211415398 podStartE2EDuration="5.211415398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:05.208511962 +0000 UTC m=+1959.149059872" watchObservedRunningTime="2026-10-29 23:15:05.211415398 +0000 UTC m=+1959.151963382" Oct 29 23:15:05.252713 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:05.252612 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8" podStartSLOduration=5.252596094 podStartE2EDuration="5.252596094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:05.251234438 +0000 UTC m=+1959.191782349" watchObservedRunningTime="2026-10-29 23:15:05.252596094 +0000 UTC m=+1959.193144005" Oct 29 23:15:05.961306 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:05.961279 4821 scope.go:122] "RemoveContainer" containerID="7e8343f852d45f97f8a0d9aa7fe7995388e2c76039af4a6d3cd8c67b55389dd6" Oct 29 23:15:05.961773 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:05.961671 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97_odh-ai-gateway-infra(514b5faa-73ee-4d54-9e6b-c474a52d82ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97" podUID="514b5faa-73ee-4d54-9e6b-c474a52d82ca" Oct 29 23:15:05.963099 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:05.963073 4821 generic.go:356] "Generic (PLEG): container finished" podID="786f75a2-8350-4081-8ca4-041ed3247252" containerID="d676204ec9fe9ff1fd07988c60a6e267ade773c07960db5cfd54ffd03ac434a1" exitCode=60 Oct 29 23:15:05.963306 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:05.963161 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf" event={"ID":"786f75a2-8350-4081-8ca4-041ed3247252","Type":"ContainerDied","Data":"d676204ec9fe9ff1fd07988c60a6e267ade773c07960db5cfd54ffd03ac434a1"} Oct 29 23:15:05.963306 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:05.963291 4821 scope.go:122] "RemoveContainer" containerID="9ad67eca7c475ad11ff85c78dc2a9a16431d9a099d3427d7dbf039e6bf3b4ada" Oct 29 23:15:05.963485 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:05.963465 4821 scope.go:122] "RemoveContainer" containerID="d676204ec9fe9ff1fd07988c60a6e267ade773c07960db5cfd54ffd03ac434a1" Oct 29 23:15:05.963850 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:05.963829 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf_odh-ai-gateway-infra(786f75a2-8350-4081-8ca4-041ed3247252)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf" podUID="786f75a2-8350-4081-8ca4-041ed3247252" Oct 29 23:15:05.965861 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:05.965842 4821 generic.go:356] "Generic (PLEG): container finished" podID="51189101-afc0-42dd-815e-a8330ec8b42a" containerID="2f0debc3f5ee1ba288366661eacb8fc64dd9144d241c53ffb83527116beea585" exitCode=60 Oct 29 23:15:05.966040 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:05.965924 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6" event={"ID":"51189101-afc0-42dd-815e-a8330ec8b42a","Type":"ContainerDied","Data":"2f0debc3f5ee1ba288366661eacb8fc64dd9144d241c53ffb83527116beea585"} Oct 29 23:15:05.966265 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:05.966244 4821 scope.go:122] "RemoveContainer" containerID="2f0debc3f5ee1ba288366661eacb8fc64dd9144d241c53ffb83527116beea585" Oct 29 23:15:05.966616 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:05.966591 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6_odh-ai-gateway-infra(51189101-afc0-42dd-815e-a8330ec8b42a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6" podUID="51189101-afc0-42dd-815e-a8330ec8b42a" Oct 29 23:15:05.968081 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:05.968061 4821 scope.go:122] "RemoveContainer" containerID="56bba4deb2101d4c06fdf4fbc5f418a3629f63c23c9c84bc0128326f006bb2ba" Oct 29 23:15:05.968411 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:05.968385 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g_odh-ai-gateway-infra(17e44a4a-8eb9-4732-8924-d76683d1e213)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g" podUID="17e44a4a-8eb9-4732-8924-d76683d1e213" Oct 29 23:15:05.969714 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:05.969684 4821 generic.go:356] "Generic (PLEG): container finished" podID="7b4b604b-3757-4c03-a745-f01e02ce82c2" containerID="a6e9a8ceb935a0091a00ba9ab03fa67799bc2509bfb0a3038ec59a6cc908580f" exitCode=60 Oct 29 23:15:05.969804 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:05.969772 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8" event={"ID":"7b4b604b-3757-4c03-a745-f01e02ce82c2","Type":"ContainerDied","Data":"a6e9a8ceb935a0091a00ba9ab03fa67799bc2509bfb0a3038ec59a6cc908580f"} Oct 29 23:15:05.970141 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:05.970125 4821 scope.go:122] "RemoveContainer" containerID="a6e9a8ceb935a0091a00ba9ab03fa67799bc2509bfb0a3038ec59a6cc908580f" Oct 29 23:15:05.970476 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:05.970449 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8_odh-ai-gateway-infra(7b4b604b-3757-4c03-a745-f01e02ce82c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8" podUID="7b4b604b-3757-4c03-a745-f01e02ce82c2" Oct 29 23:15:05.972002 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:05.971981 4821 generic.go:356] "Generic (PLEG): container finished" podID="ff81e511-3cab-4008-b2a5-1ab9774eeff4" containerID="0cf93952be4d4f8c8abc86539a5656751215c9d94f0ffef80af02fdc63b6f86a" exitCode=60 Oct 29 23:15:05.972105 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:05.972052 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp" event={"ID":"ff81e511-3cab-4008-b2a5-1ab9774eeff4","Type":"ContainerDied","Data":"0cf93952be4d4f8c8abc86539a5656751215c9d94f0ffef80af02fdc63b6f86a"} Oct 29 23:15:05.972330 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:05.972316 4821 scope.go:122] "RemoveContainer" containerID="0cf93952be4d4f8c8abc86539a5656751215c9d94f0ffef80af02fdc63b6f86a" Oct 29 23:15:05.972595 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:05.972578 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp_odh-ai-gateway-infra(ff81e511-3cab-4008-b2a5-1ab9774eeff4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp" podUID="ff81e511-3cab-4008-b2a5-1ab9774eeff4" Oct 29 23:15:05.974163 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:05.974143 4821 scope.go:122] "RemoveContainer" containerID="7e80865fe582798b8c8aa6eb4018dc99cd1072f90baa43dae6723d5c1756437f" Oct 29 23:15:05.974511 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:05.974478 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm_odh-ai-gateway-infra(2b421f85-4894-47f3-9e87-c5b683ea4e6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm" podUID="2b421f85-4894-47f3-9e87-c5b683ea4e6b" Oct 29 23:15:05.975962 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:05.975941 4821 generic.go:356] "Generic (PLEG): container finished" podID="c0219f30-2070-4cf8-bdfa-304ad5c86ad7" containerID="2068bd6ab12c5743bfcbee87a3bd07029f7abc733a738e0d2becfa9023e28f73" exitCode=60 Oct 29 23:15:05.976086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:05.975987 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5" event={"ID":"c0219f30-2070-4cf8-bdfa-304ad5c86ad7","Type":"ContainerDied","Data":"2068bd6ab12c5743bfcbee87a3bd07029f7abc733a738e0d2becfa9023e28f73"} Oct 29 23:15:05.976239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:05.976190 4821 scope.go:122] "RemoveContainer" containerID="2068bd6ab12c5743bfcbee87a3bd07029f7abc733a738e0d2becfa9023e28f73" Oct 29 23:15:05.976442 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:05.976424 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5_odh-ai-gateway-infra(c0219f30-2070-4cf8-bdfa-304ad5c86ad7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5" podUID="c0219f30-2070-4cf8-bdfa-304ad5c86ad7" Oct 29 23:15:05.979985 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:05.979967 4821 scope.go:122] "RemoveContainer" containerID="f8fc18fe03a8d0d5c9dc4009a65f7dd5cf39ec03c8de4faf1bf38d63ec80cba3" Oct 29 23:15:05.996158 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:05.996138 4821 scope.go:122] "RemoveContainer" containerID="73e258409b5d6d0525ecefcce6051d9569e8a635ebb983c75466883edff5f16e" Oct 29 23:15:06.012502 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:06.012480 4821 scope.go:122] "RemoveContainer" containerID="7ca9927aab2aef219fa8531fe6a3bb02f56639fa0047ef6a7645081680673256" Oct 29 23:15:06.028166 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:06.028143 4821 scope.go:122] "RemoveContainer" containerID="88104b40bc3ef550fd0bba169002993b97a4804ea0891ec8ccbe921911671a00" Oct 29 23:15:06.839296 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:06.839261 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5783e09-9vm4d"] Oct 29 23:15:06.840048 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:06.840031 4821 cpu_manager.go:414] "containerMap: removing container" podUID="def6461d-2480-45dc-b7cd-9f7047855ca7" containerName="revoke-keys" Oct 29 23:15:06.840119 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:06.840050 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def6461d-2480-45dc-b7cd-9f7047855ca7" containerName="revoke-keys" Oct 29 23:15:06.840189 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:06.840180 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="def6461d-2480-45dc-b7cd-9f7047855ca7" containerName="revoke-keys" Oct 29 23:15:06.845132 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:06.845108 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5783e09-9vm4d" Oct 29 23:15:06.866912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:06.866884 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5783e09-9vm4d"] Oct 29 23:15:06.872740 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:06.872701 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:15:06.963407 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:06.963378 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74bc6f99-ab50-4bd6-9b61-a8727d36f879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5783e09-9vm4d\" (UID: \"74bc6f99-ab50-4bd6-9b61-a8727d36f879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5783e09-9vm4d" Oct 29 23:15:06.963818 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:06.963510 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74bc6f99-ab50-4bd6-9b61-a8727d36f879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5783e09-9vm4d\" (UID: \"74bc6f99-ab50-4bd6-9b61-a8727d36f879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5783e09-9vm4d" Oct 29 23:15:06.963818 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:06.963717 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2xfl\" (UniqueName: \"kubernetes.io/projected/74bc6f99-ab50-4bd6-9b61-a8727d36f879-kube-api-access-b2xfl\") pod \"maas-api-revoke-sub-d5783e09-9vm4d\" (UID: \"74bc6f99-ab50-4bd6-9b61-a8727d36f879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5783e09-9vm4d" Oct 29 23:15:06.986608 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:06.986581 4821 scope.go:122] "RemoveContainer" containerID="d676204ec9fe9ff1fd07988c60a6e267ade773c07960db5cfd54ffd03ac434a1" Oct 29 23:15:06.987021 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:06.986981 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf_odh-ai-gateway-infra(786f75a2-8350-4081-8ca4-041ed3247252)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf" podUID="786f75a2-8350-4081-8ca4-041ed3247252" Oct 29 23:15:06.988440 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:06.988420 4821 scope.go:122] "RemoveContainer" containerID="2f0debc3f5ee1ba288366661eacb8fc64dd9144d241c53ffb83527116beea585" Oct 29 23:15:06.988794 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:06.988769 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6_odh-ai-gateway-infra(51189101-afc0-42dd-815e-a8330ec8b42a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6" podUID="51189101-afc0-42dd-815e-a8330ec8b42a" Oct 29 23:15:06.990101 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:06.990082 4821 scope.go:122] "RemoveContainer" containerID="a6e9a8ceb935a0091a00ba9ab03fa67799bc2509bfb0a3038ec59a6cc908580f" Oct 29 23:15:06.990460 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:06.990438 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8_odh-ai-gateway-infra(7b4b604b-3757-4c03-a745-f01e02ce82c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8" podUID="7b4b604b-3757-4c03-a745-f01e02ce82c2" Oct 29 23:15:06.991977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:06.991956 4821 scope.go:122] "RemoveContainer" containerID="0cf93952be4d4f8c8abc86539a5656751215c9d94f0ffef80af02fdc63b6f86a" Oct 29 23:15:06.992348 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:06.992309 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp_odh-ai-gateway-infra(ff81e511-3cab-4008-b2a5-1ab9774eeff4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp" podUID="ff81e511-3cab-4008-b2a5-1ab9774eeff4" Oct 29 23:15:06.994003 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:06.993984 4821 scope.go:122] "RemoveContainer" containerID="2068bd6ab12c5743bfcbee87a3bd07029f7abc733a738e0d2becfa9023e28f73" Oct 29 23:15:06.994404 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:06.994377 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5_odh-ai-gateway-infra(c0219f30-2070-4cf8-bdfa-304ad5c86ad7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5" podUID="c0219f30-2070-4cf8-bdfa-304ad5c86ad7" Oct 29 23:15:07.065312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:07.065279 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2xfl\" (UniqueName: \"kubernetes.io/projected/74bc6f99-ab50-4bd6-9b61-a8727d36f879-kube-api-access-b2xfl\") pod \"maas-api-revoke-sub-d5783e09-9vm4d\" (UID: \"74bc6f99-ab50-4bd6-9b61-a8727d36f879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5783e09-9vm4d" Oct 29 23:15:07.065571 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:07.065552 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74bc6f99-ab50-4bd6-9b61-a8727d36f879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5783e09-9vm4d\" (UID: \"74bc6f99-ab50-4bd6-9b61-a8727d36f879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5783e09-9vm4d" Oct 29 23:15:07.065707 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:07.065691 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74bc6f99-ab50-4bd6-9b61-a8727d36f879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5783e09-9vm4d\" (UID: \"74bc6f99-ab50-4bd6-9b61-a8727d36f879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5783e09-9vm4d" Oct 29 23:15:07.066174 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:07.066149 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74bc6f99-ab50-4bd6-9b61-a8727d36f879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5783e09-9vm4d\" (UID: \"74bc6f99-ab50-4bd6-9b61-a8727d36f879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5783e09-9vm4d" Oct 29 23:15:07.068285 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:07.068264 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74bc6f99-ab50-4bd6-9b61-a8727d36f879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5783e09-9vm4d\" (UID: \"74bc6f99-ab50-4bd6-9b61-a8727d36f879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5783e09-9vm4d" Oct 29 23:15:07.091016 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:07.090950 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2xfl\" (UniqueName: \"kubernetes.io/projected/74bc6f99-ab50-4bd6-9b61-a8727d36f879-kube-api-access-b2xfl\") pod \"maas-api-revoke-sub-d5783e09-9vm4d\" (UID: \"74bc6f99-ab50-4bd6-9b61-a8727d36f879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5783e09-9vm4d" Oct 29 23:15:07.160067 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:07.160035 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5783e09-9vm4d" Oct 29 23:15:07.476011 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:15:07.475981 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74bc6f99_ab50_4bd6_9b61_a8727d36f879.slice/crio-5f644387d15dd485a1503ea29ddb1cc8fdbe53ba91a3ae3d67400d1731f540f8 WatchSource:0}: Error finding container 5f644387d15dd485a1503ea29ddb1cc8fdbe53ba91a3ae3d67400d1731f540f8: Status 404 returned error can't find the container with id 5f644387d15dd485a1503ea29ddb1cc8fdbe53ba91a3ae3d67400d1731f540f8 Oct 29 23:15:07.477022 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:07.476996 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5783e09-9vm4d"] Oct 29 23:15:08.013758 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:08.013701 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5783e09-9vm4d" event={"ID":"74bc6f99-ab50-4bd6-9b61-a8727d36f879","Type":"ContainerStarted","Data":"5f644387d15dd485a1503ea29ddb1cc8fdbe53ba91a3ae3d67400d1731f540f8"} Oct 29 23:15:08.083933 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:08.083896 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b"] Oct 29 23:15:08.084622 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:08.084585 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" podUID="137c7cf5-e922-49d1-acbc-fc0b13899da5" containerName="istio-proxy" containerID="cri-o://5fa4fac44d9dc85377625cb0f50c6c1f1039ed1b104e3ed81de01b4c0e7b3eca" gracePeriod=30 Oct 29 23:15:08.094376 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:08.094347 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr"] Oct 29 23:15:08.094591 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:08.094564 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" podUID="c0da6fd9-efd5-4587-b764-524010b6a352" containerName="istio-proxy" containerID="cri-o://509e9a78c01a9d34f3ca79be700e171ac50110f12c53a9e10b3d2fa1181d25a2" gracePeriod=30 Oct 29 23:15:09.024890 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:09.024847 4821 generic.go:356] "Generic (PLEG): container finished" podID="74bc6f99-ab50-4bd6-9b61-a8727d36f879" containerID="6b77984258c52ea3cd87ac5ecbaf4b39967292b7a6cb556516084b01aca652d7" exitCode=0 Oct 29 23:15:09.025393 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:09.024933 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5783e09-9vm4d" event={"ID":"74bc6f99-ab50-4bd6-9b61-a8727d36f879","Type":"ContainerDied","Data":"6b77984258c52ea3cd87ac5ecbaf4b39967292b7a6cb556516084b01aca652d7"} Oct 29 23:15:09.955923 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:09.955883 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" Oct 29 23:15:10.257876 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:10.257848 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5783e09-9vm4d" Oct 29 23:15:10.438972 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:10.438938 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74bc6f99-ab50-4bd6-9b61-a8727d36f879-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74bc6f99-ab50-4bd6-9b61-a8727d36f879-maas-api-openshift-ca\") pod \"74bc6f99-ab50-4bd6-9b61-a8727d36f879\" (UID: \"74bc6f99-ab50-4bd6-9b61-a8727d36f879\") " Oct 29 23:15:10.439307 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:10.439036 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74bc6f99-ab50-4bd6-9b61-a8727d36f879-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74bc6f99-ab50-4bd6-9b61-a8727d36f879-maas-api-xks-ca\") pod \"74bc6f99-ab50-4bd6-9b61-a8727d36f879\" (UID: \"74bc6f99-ab50-4bd6-9b61-a8727d36f879\") " Oct 29 23:15:10.439307 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:10.439118 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74bc6f99-ab50-4bd6-9b61-a8727d36f879-kube-api-access-b2xfl\" (UniqueName: \"kubernetes.io/projected/74bc6f99-ab50-4bd6-9b61-a8727d36f879-kube-api-access-b2xfl\") pod \"74bc6f99-ab50-4bd6-9b61-a8727d36f879\" (UID: \"74bc6f99-ab50-4bd6-9b61-a8727d36f879\") " Oct 29 23:15:10.439537 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:10.439319 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74bc6f99-ab50-4bd6-9b61-a8727d36f879-maas-api-openshift-ca" pod "74bc6f99-ab50-4bd6-9b61-a8727d36f879" (UID: "74bc6f99-ab50-4bd6-9b61-a8727d36f879"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:10.439942 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:10.439914 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74bc6f99-ab50-4bd6-9b61-a8727d36f879-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:10.441472 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:10.441444 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74bc6f99-ab50-4bd6-9b61-a8727d36f879-maas-api-xks-ca" pod "74bc6f99-ab50-4bd6-9b61-a8727d36f879" (UID: "74bc6f99-ab50-4bd6-9b61-a8727d36f879"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:15:10.441577 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:10.441555 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74bc6f99-ab50-4bd6-9b61-a8727d36f879-kube-api-access-b2xfl" pod "74bc6f99-ab50-4bd6-9b61-a8727d36f879" (UID: "74bc6f99-ab50-4bd6-9b61-a8727d36f879"). InnerVolumeSpecName "kube-api-access-b2xfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:10.541366 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:10.541286 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2xfl\" (UniqueName: \"kubernetes.io/projected/74bc6f99-ab50-4bd6-9b61-a8727d36f879-kube-api-access-b2xfl\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:10.541366 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:10.541315 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74bc6f99-ab50-4bd6-9b61-a8727d36f879-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:11.048560 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:11.048528 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5783e09-9vm4d" event={"ID":"74bc6f99-ab50-4bd6-9b61-a8727d36f879","Type":"ContainerDied","Data":"5f644387d15dd485a1503ea29ddb1cc8fdbe53ba91a3ae3d67400d1731f540f8"} Oct 29 23:15:11.048749 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:11.048563 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5783e09-9vm4d" Oct 29 23:15:11.048749 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:11.048577 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f644387d15dd485a1503ea29ddb1cc8fdbe53ba91a3ae3d67400d1731f540f8" Oct 29 23:15:13.477395 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.477359 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:15:13.514396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.514372 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:15:13.593538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.592813 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-workload-certs\") pod \"137c7cf5-e922-49d1-acbc-fc0b13899da5\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " Oct 29 23:15:13.593538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.592886 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-istio-envoy\") pod \"c0da6fd9-efd5-4587-b764-524010b6a352\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " Oct 29 23:15:13.593538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.593011 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0da6fd9-efd5-4587-b764-524010b6a352-kube-api-access-lfv6g\" (UniqueName: \"kubernetes.io/projected/c0da6fd9-efd5-4587-b764-524010b6a352-kube-api-access-lfv6g\") pod \"c0da6fd9-efd5-4587-b764-524010b6a352\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " Oct 29 23:15:13.593538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.593051 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-envoy\") pod \"137c7cf5-e922-49d1-acbc-fc0b13899da5\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " Oct 29 23:15:13.593538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.593077 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0da6fd9-efd5-4587-b764-524010b6a352-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0da6fd9-efd5-4587-b764-524010b6a352-istiod-ca-cert\") pod \"c0da6fd9-efd5-4587-b764-524010b6a352\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " Oct 29 23:15:13.593538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.593155 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0da6fd9-efd5-4587-b764-524010b6a352-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0da6fd9-efd5-4587-b764-524010b6a352-istio-podinfo\") pod \"c0da6fd9-efd5-4587-b764-524010b6a352\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " Oct 29 23:15:13.593538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.593189 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/137c7cf5-e922-49d1-acbc-fc0b13899da5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137c7cf5-e922-49d1-acbc-fc0b13899da5-istiod-ca-cert\") pod \"137c7cf5-e922-49d1-acbc-fc0b13899da5\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " Oct 29 23:15:13.593538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.593312 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-credential-socket\") pod \"c0da6fd9-efd5-4587-b764-524010b6a352\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " Oct 29 23:15:13.593538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.593353 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-data\") pod \"137c7cf5-e922-49d1-acbc-fc0b13899da5\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " Oct 29 23:15:13.593538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.593389 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-workload-socket\") pod \"c0da6fd9-efd5-4587-b764-524010b6a352\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " Oct 29 23:15:13.593538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.593422 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-istio-data\") pod \"c0da6fd9-efd5-4587-b764-524010b6a352\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " Oct 29 23:15:13.594450 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.593557 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-workload-socket\") pod \"137c7cf5-e922-49d1-acbc-fc0b13899da5\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " Oct 29 23:15:13.594450 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.593642 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0da6fd9-efd5-4587-b764-524010b6a352-istio-token\" (UniqueName: \"kubernetes.io/projected/c0da6fd9-efd5-4587-b764-524010b6a352-istio-token\") pod \"c0da6fd9-efd5-4587-b764-524010b6a352\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " Oct 29 23:15:13.594450 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.593704 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-credential-socket\") pod \"137c7cf5-e922-49d1-acbc-fc0b13899da5\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " Oct 29 23:15:13.594450 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.593730 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/137c7cf5-e922-49d1-acbc-fc0b13899da5-kube-api-access-fblfg\" (UniqueName: \"kubernetes.io/projected/137c7cf5-e922-49d1-acbc-fc0b13899da5-kube-api-access-fblfg\") pod \"137c7cf5-e922-49d1-acbc-fc0b13899da5\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " Oct 29 23:15:13.594450 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.593808 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-token\" (UniqueName: \"kubernetes.io/projected/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-token\") pod \"137c7cf5-e922-49d1-acbc-fc0b13899da5\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " Oct 29 23:15:13.594450 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.593859 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-podinfo\") pod \"137c7cf5-e922-49d1-acbc-fc0b13899da5\" (UID: \"137c7cf5-e922-49d1-acbc-fc0b13899da5\") " Oct 29 23:15:13.594450 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.593888 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-workload-certs\") pod \"c0da6fd9-efd5-4587-b764-524010b6a352\" (UID: \"c0da6fd9-efd5-4587-b764-524010b6a352\") " Oct 29 23:15:13.594867 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.594702 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-data" pod "137c7cf5-e922-49d1-acbc-fc0b13899da5" (UID: "137c7cf5-e922-49d1-acbc-fc0b13899da5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:13.594950 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.594924 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-credential-socket" pod "c0da6fd9-efd5-4587-b764-524010b6a352" (UID: "c0da6fd9-efd5-4587-b764-524010b6a352"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:13.595173 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.595152 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-istio-data" pod "c0da6fd9-efd5-4587-b764-524010b6a352" (UID: "c0da6fd9-efd5-4587-b764-524010b6a352"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:13.595722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.595462 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0da6fd9-efd5-4587-b764-524010b6a352-istiod-ca-cert" pod "c0da6fd9-efd5-4587-b764-524010b6a352" (UID: "c0da6fd9-efd5-4587-b764-524010b6a352"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:13.596218 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.596087 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137c7cf5-e922-49d1-acbc-fc0b13899da5-istiod-ca-cert" pod "137c7cf5-e922-49d1-acbc-fc0b13899da5" (UID: "137c7cf5-e922-49d1-acbc-fc0b13899da5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:13.596816 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.596644 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-workload-certs" pod "137c7cf5-e922-49d1-acbc-fc0b13899da5" (UID: "137c7cf5-e922-49d1-acbc-fc0b13899da5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:13.597260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.596962 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-workload-socket" pod "137c7cf5-e922-49d1-acbc-fc0b13899da5" (UID: "137c7cf5-e922-49d1-acbc-fc0b13899da5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:13.597657 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.597599 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-credential-socket" pod "137c7cf5-e922-49d1-acbc-fc0b13899da5" (UID: "137c7cf5-e922-49d1-acbc-fc0b13899da5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:13.597770 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.597641 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-workload-socket" pod "c0da6fd9-efd5-4587-b764-524010b6a352" (UID: "c0da6fd9-efd5-4587-b764-524010b6a352"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:13.598375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.598351 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-workload-certs" pod "c0da6fd9-efd5-4587-b764-524010b6a352" (UID: "c0da6fd9-efd5-4587-b764-524010b6a352"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:13.598682 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.598636 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0da6fd9-efd5-4587-b764-524010b6a352-kube-api-access-lfv6g" pod "c0da6fd9-efd5-4587-b764-524010b6a352" (UID: "c0da6fd9-efd5-4587-b764-524010b6a352"). InnerVolumeSpecName "kube-api-access-lfv6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:13.599336 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.599315 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfv6g\" (UniqueName: \"kubernetes.io/projected/c0da6fd9-efd5-4587-b764-524010b6a352-kube-api-access-lfv6g\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:13.599565 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.599538 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0da6fd9-efd5-4587-b764-524010b6a352-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:13.599565 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.599567 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137c7cf5-e922-49d1-acbc-fc0b13899da5-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:13.599746 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.599580 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:13.599746 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.599594 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:13.599746 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.599607 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:13.599746 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.599620 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:13.599746 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.599633 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:13.599746 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.599646 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:13.599746 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.599660 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:13.599746 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.599676 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:13.604558 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.604528 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-envoy" pod "137c7cf5-e922-49d1-acbc-fc0b13899da5" (UID: "137c7cf5-e922-49d1-acbc-fc0b13899da5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:13.604680 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.604654 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137c7cf5-e922-49d1-acbc-fc0b13899da5-kube-api-access-fblfg" pod "137c7cf5-e922-49d1-acbc-fc0b13899da5" (UID: "137c7cf5-e922-49d1-acbc-fc0b13899da5"). InnerVolumeSpecName "kube-api-access-fblfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:13.604805 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.604781 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-istio-envoy" pod "c0da6fd9-efd5-4587-b764-524010b6a352" (UID: "c0da6fd9-efd5-4587-b764-524010b6a352"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:13.604892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.604873 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0da6fd9-efd5-4587-b764-524010b6a352-istio-podinfo" pod "c0da6fd9-efd5-4587-b764-524010b6a352" (UID: "c0da6fd9-efd5-4587-b764-524010b6a352"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:15:13.604996 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.604979 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-token" pod "137c7cf5-e922-49d1-acbc-fc0b13899da5" (UID: "137c7cf5-e922-49d1-acbc-fc0b13899da5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:13.606496 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.606461 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0da6fd9-efd5-4587-b764-524010b6a352-istio-token" pod "c0da6fd9-efd5-4587-b764-524010b6a352" (UID: "c0da6fd9-efd5-4587-b764-524010b6a352"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:13.607013 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.606986 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-podinfo" pod "137c7cf5-e922-49d1-acbc-fc0b13899da5" (UID: "137c7cf5-e922-49d1-acbc-fc0b13899da5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:15:13.701372 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.701290 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0da6fd9-efd5-4587-b764-524010b6a352-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:13.701372 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.701323 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0da6fd9-efd5-4587-b764-524010b6a352-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:13.701372 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.701338 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fblfg\" (UniqueName: \"kubernetes.io/projected/137c7cf5-e922-49d1-acbc-fc0b13899da5-kube-api-access-fblfg\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:13.701372 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.701354 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:13.701372 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.701367 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:13.701700 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.701379 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0da6fd9-efd5-4587-b764-524010b6a352-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:13.701700 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:13.701393 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137c7cf5-e922-49d1-acbc-fc0b13899da5-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:14.079079 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:14.078979 4821 generic.go:356] "Generic (PLEG): container finished" podID="c0da6fd9-efd5-4587-b764-524010b6a352" containerID="509e9a78c01a9d34f3ca79be700e171ac50110f12c53a9e10b3d2fa1181d25a2" exitCode=0 Oct 29 23:15:14.079079 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:14.079040 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" Oct 29 23:15:14.079361 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:14.079080 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" event={"ID":"c0da6fd9-efd5-4587-b764-524010b6a352","Type":"ContainerDied","Data":"509e9a78c01a9d34f3ca79be700e171ac50110f12c53a9e10b3d2fa1181d25a2"} Oct 29 23:15:14.079361 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:14.079166 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr" event={"ID":"c0da6fd9-efd5-4587-b764-524010b6a352","Type":"ContainerDied","Data":"15c3d6a5e3c7d82f14fac836c1b354494a22a2ec127854ced69495706f25eb31"} Oct 29 23:15:14.079361 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:14.079228 4821 scope.go:122] "RemoveContainer" containerID="509e9a78c01a9d34f3ca79be700e171ac50110f12c53a9e10b3d2fa1181d25a2" Oct 29 23:15:14.082064 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:14.082037 4821 generic.go:356] "Generic (PLEG): container finished" podID="137c7cf5-e922-49d1-acbc-fc0b13899da5" containerID="5fa4fac44d9dc85377625cb0f50c6c1f1039ed1b104e3ed81de01b4c0e7b3eca" exitCode=0 Oct 29 23:15:14.082323 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:14.082251 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" event={"ID":"137c7cf5-e922-49d1-acbc-fc0b13899da5","Type":"ContainerDied","Data":"5fa4fac44d9dc85377625cb0f50c6c1f1039ed1b104e3ed81de01b4c0e7b3eca"} Oct 29 23:15:14.082323 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:14.082302 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" event={"ID":"137c7cf5-e922-49d1-acbc-fc0b13899da5","Type":"ContainerDied","Data":"5ef2c6b4f114e5c2c65856558c3f5ec55c9ab0963a51134df511389410eea6af"} Oct 29 23:15:14.082509 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:14.082376 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b" Oct 29 23:15:14.098140 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:14.098110 4821 scope.go:122] "RemoveContainer" containerID="509e9a78c01a9d34f3ca79be700e171ac50110f12c53a9e10b3d2fa1181d25a2" Oct 29 23:15:14.098552 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:14.098512 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"509e9a78c01a9d34f3ca79be700e171ac50110f12c53a9e10b3d2fa1181d25a2\": container with ID starting with 509e9a78c01a9d34f3ca79be700e171ac50110f12c53a9e10b3d2fa1181d25a2 not found: ID does not exist" containerID="509e9a78c01a9d34f3ca79be700e171ac50110f12c53a9e10b3d2fa1181d25a2" Oct 29 23:15:14.098680 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:14.098553 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509e9a78c01a9d34f3ca79be700e171ac50110f12c53a9e10b3d2fa1181d25a2"} err="failed to get container status \"509e9a78c01a9d34f3ca79be700e171ac50110f12c53a9e10b3d2fa1181d25a2\": rpc error: code = NotFound desc = could not find container \"509e9a78c01a9d34f3ca79be700e171ac50110f12c53a9e10b3d2fa1181d25a2\": container with ID starting with 509e9a78c01a9d34f3ca79be700e171ac50110f12c53a9e10b3d2fa1181d25a2 not found: ID does not exist" Oct 29 23:15:14.098680 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:14.098581 4821 scope.go:122] "RemoveContainer" containerID="5fa4fac44d9dc85377625cb0f50c6c1f1039ed1b104e3ed81de01b4c0e7b3eca" Oct 29 23:15:14.138601 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:14.138581 4821 scope.go:122] "RemoveContainer" containerID="5fa4fac44d9dc85377625cb0f50c6c1f1039ed1b104e3ed81de01b4c0e7b3eca" Oct 29 23:15:14.138918 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:14.138896 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fa4fac44d9dc85377625cb0f50c6c1f1039ed1b104e3ed81de01b4c0e7b3eca\": container with ID starting with 5fa4fac44d9dc85377625cb0f50c6c1f1039ed1b104e3ed81de01b4c0e7b3eca not found: ID does not exist" containerID="5fa4fac44d9dc85377625cb0f50c6c1f1039ed1b104e3ed81de01b4c0e7b3eca" Oct 29 23:15:14.139006 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:14.138923 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fa4fac44d9dc85377625cb0f50c6c1f1039ed1b104e3ed81de01b4c0e7b3eca"} err="failed to get container status \"5fa4fac44d9dc85377625cb0f50c6c1f1039ed1b104e3ed81de01b4c0e7b3eca\": rpc error: code = NotFound desc = could not find container \"5fa4fac44d9dc85377625cb0f50c6c1f1039ed1b104e3ed81de01b4c0e7b3eca\": container with ID starting with 5fa4fac44d9dc85377625cb0f50c6c1f1039ed1b104e3ed81de01b4c0e7b3eca not found: ID does not exist" Oct 29 23:15:14.150827 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:14.150807 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b"] Oct 29 23:15:14.155490 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:14.155468 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-9p68b"] Oct 29 23:15:14.183567 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:14.183534 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="137c7cf5-e922-49d1-acbc-fc0b13899da5" path="/var/lib/kubelet/pods/137c7cf5-e922-49d1-acbc-fc0b13899da5/volumes" Oct 29 23:15:14.190840 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:14.190816 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr"] Oct 29 23:15:14.195379 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:14.195355 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-56kxr"] Oct 29 23:15:16.168524 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:16.168494 4821 scope.go:122] "RemoveContainer" containerID="56bba4deb2101d4c06fdf4fbc5f418a3629f63c23c9c84bc0128326f006bb2ba" Oct 29 23:15:16.169024 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:16.168656 4821 scope.go:122] "RemoveContainer" containerID="7e8343f852d45f97f8a0d9aa7fe7995388e2c76039af4a6d3cd8c67b55389dd6" Oct 29 23:15:16.169024 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:16.168814 4821 scope.go:122] "RemoveContainer" containerID="7e80865fe582798b8c8aa6eb4018dc99cd1072f90baa43dae6723d5c1756437f" Oct 29 23:15:16.174230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:16.174171 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0da6fd9-efd5-4587-b764-524010b6a352" path="/var/lib/kubelet/pods/c0da6fd9-efd5-4587-b764-524010b6a352/volumes" Oct 29 23:15:17.116002 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:17.115966 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g" event={"ID":"17e44a4a-8eb9-4732-8924-d76683d1e213","Type":"ContainerStarted","Data":"0985436bd6ced5dba5a38f6ff7b5dc6c20b0cf7090515b0fc5e79ad3c180ddf4"} Oct 29 23:15:17.118758 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:17.118727 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm" event={"ID":"2b421f85-4894-47f3-9e87-c5b683ea4e6b","Type":"ContainerStarted","Data":"ba980c5a021f4316289ecc6a44b8787d50ae7ae72103619dce324888870c801e"} Oct 29 23:15:17.120994 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:17.120972 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97" event={"ID":"514b5faa-73ee-4d54-9e6b-c474a52d82ca","Type":"ContainerStarted","Data":"f7ee647f314e0168543f69e92698bd03781e58630ce228c136d13c6fc2a62986"} Oct 29 23:15:17.169711 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:17.164189 4821 scope.go:122] "RemoveContainer" containerID="2f0debc3f5ee1ba288366661eacb8fc64dd9144d241c53ffb83527116beea585" Oct 29 23:15:18.132152 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:18.132118 4821 generic.go:356] "Generic (PLEG): container finished" podID="2b421f85-4894-47f3-9e87-c5b683ea4e6b" containerID="ba980c5a021f4316289ecc6a44b8787d50ae7ae72103619dce324888870c801e" exitCode=60 Oct 29 23:15:18.132369 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:18.132179 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm" event={"ID":"2b421f85-4894-47f3-9e87-c5b683ea4e6b","Type":"ContainerDied","Data":"ba980c5a021f4316289ecc6a44b8787d50ae7ae72103619dce324888870c801e"} Oct 29 23:15:18.132369 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:18.132280 4821 scope.go:122] "RemoveContainer" containerID="7e80865fe582798b8c8aa6eb4018dc99cd1072f90baa43dae6723d5c1756437f" Oct 29 23:15:18.132730 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:18.132710 4821 scope.go:122] "RemoveContainer" containerID="ba980c5a021f4316289ecc6a44b8787d50ae7ae72103619dce324888870c801e" Oct 29 23:15:18.133129 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:18.133103 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm_odh-ai-gateway-infra(2b421f85-4894-47f3-9e87-c5b683ea4e6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm" podUID="2b421f85-4894-47f3-9e87-c5b683ea4e6b" Oct 29 23:15:18.134666 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:18.134648 4821 generic.go:356] "Generic (PLEG): container finished" podID="514b5faa-73ee-4d54-9e6b-c474a52d82ca" containerID="f7ee647f314e0168543f69e92698bd03781e58630ce228c136d13c6fc2a62986" exitCode=60 Oct 29 23:15:18.134754 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:18.134714 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97" event={"ID":"514b5faa-73ee-4d54-9e6b-c474a52d82ca","Type":"ContainerDied","Data":"f7ee647f314e0168543f69e92698bd03781e58630ce228c136d13c6fc2a62986"} Oct 29 23:15:18.135054 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:18.135039 4821 scope.go:122] "RemoveContainer" containerID="f7ee647f314e0168543f69e92698bd03781e58630ce228c136d13c6fc2a62986" Oct 29 23:15:18.135488 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:18.135414 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97_odh-ai-gateway-infra(514b5faa-73ee-4d54-9e6b-c474a52d82ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97" podUID="514b5faa-73ee-4d54-9e6b-c474a52d82ca" Oct 29 23:15:18.137026 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:18.137005 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6" event={"ID":"51189101-afc0-42dd-815e-a8330ec8b42a","Type":"ContainerStarted","Data":"00047ce3ca85fdd758875f264d75d5c1ffccca3fce2571503d7389f93c8c5f83"} Oct 29 23:15:18.139438 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:18.139418 4821 generic.go:356] "Generic (PLEG): container finished" podID="17e44a4a-8eb9-4732-8924-d76683d1e213" containerID="0985436bd6ced5dba5a38f6ff7b5dc6c20b0cf7090515b0fc5e79ad3c180ddf4" exitCode=60 Oct 29 23:15:18.139531 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:18.139445 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g" event={"ID":"17e44a4a-8eb9-4732-8924-d76683d1e213","Type":"ContainerDied","Data":"0985436bd6ced5dba5a38f6ff7b5dc6c20b0cf7090515b0fc5e79ad3c180ddf4"} Oct 29 23:15:18.139758 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:18.139745 4821 scope.go:122] "RemoveContainer" containerID="0985436bd6ced5dba5a38f6ff7b5dc6c20b0cf7090515b0fc5e79ad3c180ddf4" Oct 29 23:15:18.140063 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:18.140047 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g_odh-ai-gateway-infra(17e44a4a-8eb9-4732-8924-d76683d1e213)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g" podUID="17e44a4a-8eb9-4732-8924-d76683d1e213" Oct 29 23:15:18.148509 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:18.148492 4821 scope.go:122] "RemoveContainer" containerID="7e8343f852d45f97f8a0d9aa7fe7995388e2c76039af4a6d3cd8c67b55389dd6" Oct 29 23:15:18.162716 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:18.162697 4821 scope.go:122] "RemoveContainer" containerID="56bba4deb2101d4c06fdf4fbc5f418a3629f63c23c9c84bc0128326f006bb2ba" Oct 29 23:15:18.162879 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:18.162861 4821 scope.go:122] "RemoveContainer" containerID="2068bd6ab12c5743bfcbee87a3bd07029f7abc733a738e0d2becfa9023e28f73" Oct 29 23:15:18.163006 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:18.162988 4821 scope.go:122] "RemoveContainer" containerID="0cf93952be4d4f8c8abc86539a5656751215c9d94f0ffef80af02fdc63b6f86a" Oct 29 23:15:18.163081 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:18.163062 4821 scope.go:122] "RemoveContainer" containerID="d676204ec9fe9ff1fd07988c60a6e267ade773c07960db5cfd54ffd03ac434a1" Oct 29 23:15:18.177931 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:18.177908 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g"] Oct 29 23:15:18.214130 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:18.214105 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97"] Oct 29 23:15:18.276543 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:18.276507 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm"] Oct 29 23:15:19.152108 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.152076 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf" event={"ID":"786f75a2-8350-4081-8ca4-041ed3247252","Type":"ContainerStarted","Data":"15057460d6abe1583fc1458c6cf762505f6550ed82b3de7c483516c859535c3b"} Oct 29 23:15:19.154658 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.154626 4821 generic.go:356] "Generic (PLEG): container finished" podID="51189101-afc0-42dd-815e-a8330ec8b42a" containerID="00047ce3ca85fdd758875f264d75d5c1ffccca3fce2571503d7389f93c8c5f83" exitCode=60 Oct 29 23:15:19.154765 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.154695 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6" event={"ID":"51189101-afc0-42dd-815e-a8330ec8b42a","Type":"ContainerDied","Data":"00047ce3ca85fdd758875f264d75d5c1ffccca3fce2571503d7389f93c8c5f83"} Oct 29 23:15:19.154765 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.154739 4821 scope.go:122] "RemoveContainer" containerID="2f0debc3f5ee1ba288366661eacb8fc64dd9144d241c53ffb83527116beea585" Oct 29 23:15:19.155112 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.155097 4821 scope.go:122] "RemoveContainer" containerID="00047ce3ca85fdd758875f264d75d5c1ffccca3fce2571503d7389f93c8c5f83" Oct 29 23:15:19.155516 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:19.155496 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6_odh-ai-gateway-infra(51189101-afc0-42dd-815e-a8330ec8b42a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6" podUID="51189101-afc0-42dd-815e-a8330ec8b42a" Oct 29 23:15:19.159522 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.159257 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp" event={"ID":"ff81e511-3cab-4008-b2a5-1ab9774eeff4","Type":"ContainerStarted","Data":"0967ff9f9dba158188845b788afa3558b174e2d7a1f8231cb31fed0683fff04c"} Oct 29 23:15:19.163717 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.163690 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5" event={"ID":"c0219f30-2070-4cf8-bdfa-304ad5c86ad7","Type":"ContainerStarted","Data":"dd6b8d7c5081a3b582abfb2b3c62211305bee5c03bd6e71eafc2c3da1fe621cd"} Oct 29 23:15:19.325103 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.325071 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6"] Oct 29 23:15:19.483555 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.483527 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97" Oct 29 23:15:19.490504 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.490476 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g" Oct 29 23:15:19.495580 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.495556 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm" Oct 29 23:15:19.625994 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.625957 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b421f85-4894-47f3-9e87-c5b683ea4e6b-kube-api-access-8vnjr\" (UniqueName: \"kubernetes.io/projected/2b421f85-4894-47f3-9e87-c5b683ea4e6b-kube-api-access-8vnjr\") pod \"2b421f85-4894-47f3-9e87-c5b683ea4e6b\" (UID: \"2b421f85-4894-47f3-9e87-c5b683ea4e6b\") " Oct 29 23:15:19.626151 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.626089 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e44a4a-8eb9-4732-8924-d76683d1e213-kube-api-access-pskb6\" (UniqueName: \"kubernetes.io/projected/17e44a4a-8eb9-4732-8924-d76683d1e213-kube-api-access-pskb6\") pod \"17e44a4a-8eb9-4732-8924-d76683d1e213\" (UID: \"17e44a4a-8eb9-4732-8924-d76683d1e213\") " Oct 29 23:15:19.626270 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.626173 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/514b5faa-73ee-4d54-9e6b-c474a52d82ca-kube-api-access-6mkmc\" (UniqueName: \"kubernetes.io/projected/514b5faa-73ee-4d54-9e6b-c474a52d82ca-kube-api-access-6mkmc\") pod \"514b5faa-73ee-4d54-9e6b-c474a52d82ca\" (UID: \"514b5faa-73ee-4d54-9e6b-c474a52d82ca\") " Oct 29 23:15:19.626270 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.626248 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17e44a4a-8eb9-4732-8924-d76683d1e213-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e44a4a-8eb9-4732-8924-d76683d1e213-maas-api-service-ca\") pod \"17e44a4a-8eb9-4732-8924-d76683d1e213\" (UID: \"17e44a4a-8eb9-4732-8924-d76683d1e213\") " Oct 29 23:15:19.626495 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.626473 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b421f85-4894-47f3-9e87-c5b683ea4e6b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b421f85-4894-47f3-9e87-c5b683ea4e6b-maas-api-service-ca\") pod \"2b421f85-4894-47f3-9e87-c5b683ea4e6b\" (UID: \"2b421f85-4894-47f3-9e87-c5b683ea4e6b\") " Oct 29 23:15:19.626576 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.626561 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/514b5faa-73ee-4d54-9e6b-c474a52d82ca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/514b5faa-73ee-4d54-9e6b-c474a52d82ca-maas-api-service-ca\") pod \"514b5faa-73ee-4d54-9e6b-c474a52d82ca\" (UID: \"514b5faa-73ee-4d54-9e6b-c474a52d82ca\") " Oct 29 23:15:19.628077 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.628042 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e44a4a-8eb9-4732-8924-d76683d1e213-maas-api-service-ca" pod "17e44a4a-8eb9-4732-8924-d76683d1e213" (UID: "17e44a4a-8eb9-4732-8924-d76683d1e213"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:19.628286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.628190 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514b5faa-73ee-4d54-9e6b-c474a52d82ca-maas-api-service-ca" pod "514b5faa-73ee-4d54-9e6b-c474a52d82ca" (UID: "514b5faa-73ee-4d54-9e6b-c474a52d82ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:19.628392 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.628380 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b421f85-4894-47f3-9e87-c5b683ea4e6b-maas-api-service-ca" pod "2b421f85-4894-47f3-9e87-c5b683ea4e6b" (UID: "2b421f85-4894-47f3-9e87-c5b683ea4e6b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:19.631616 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.631311 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e44a4a-8eb9-4732-8924-d76683d1e213-kube-api-access-pskb6" pod "17e44a4a-8eb9-4732-8924-d76683d1e213" (UID: "17e44a4a-8eb9-4732-8924-d76683d1e213"). InnerVolumeSpecName "kube-api-access-pskb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:19.635607 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.635570 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514b5faa-73ee-4d54-9e6b-c474a52d82ca-kube-api-access-6mkmc" pod "514b5faa-73ee-4d54-9e6b-c474a52d82ca" (UID: "514b5faa-73ee-4d54-9e6b-c474a52d82ca"). InnerVolumeSpecName "kube-api-access-6mkmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:19.636113 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.636081 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b421f85-4894-47f3-9e87-c5b683ea4e6b-kube-api-access-8vnjr" pod "2b421f85-4894-47f3-9e87-c5b683ea4e6b" (UID: "2b421f85-4894-47f3-9e87-c5b683ea4e6b"). InnerVolumeSpecName "kube-api-access-8vnjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:19.729140 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.729053 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mkmc\" (UniqueName: \"kubernetes.io/projected/514b5faa-73ee-4d54-9e6b-c474a52d82ca-kube-api-access-6mkmc\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:19.729140 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.729093 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e44a4a-8eb9-4732-8924-d76683d1e213-maas-api-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:19.729140 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.729108 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b421f85-4894-47f3-9e87-c5b683ea4e6b-maas-api-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:19.729140 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.729122 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/514b5faa-73ee-4d54-9e6b-c474a52d82ca-maas-api-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:19.729140 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.729137 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vnjr\" (UniqueName: \"kubernetes.io/projected/2b421f85-4894-47f3-9e87-c5b683ea4e6b-kube-api-access-8vnjr\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:19.729449 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:19.729151 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pskb6\" (UniqueName: \"kubernetes.io/projected/17e44a4a-8eb9-4732-8924-d76683d1e213-kube-api-access-pskb6\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:20.174253 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.174069 4821 scope.go:122] "RemoveContainer" containerID="a6e9a8ceb935a0091a00ba9ab03fa67799bc2509bfb0a3038ec59a6cc908580f" Oct 29 23:15:20.201362 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.201323 4821 generic.go:356] "Generic (PLEG): container finished" podID="ff81e511-3cab-4008-b2a5-1ab9774eeff4" containerID="0967ff9f9dba158188845b788afa3558b174e2d7a1f8231cb31fed0683fff04c" exitCode=60 Oct 29 23:15:20.202357 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.202320 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp" event={"ID":"ff81e511-3cab-4008-b2a5-1ab9774eeff4","Type":"ContainerDied","Data":"0967ff9f9dba158188845b788afa3558b174e2d7a1f8231cb31fed0683fff04c"} Oct 29 23:15:20.202498 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.202395 4821 scope.go:122] "RemoveContainer" containerID="0cf93952be4d4f8c8abc86539a5656751215c9d94f0ffef80af02fdc63b6f86a" Oct 29 23:15:20.203024 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.202966 4821 scope.go:122] "RemoveContainer" containerID="0967ff9f9dba158188845b788afa3558b174e2d7a1f8231cb31fed0683fff04c" Oct 29 23:15:20.203541 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:20.203513 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp_odh-ai-gateway-infra(ff81e511-3cab-4008-b2a5-1ab9774eeff4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp" podUID="ff81e511-3cab-4008-b2a5-1ab9774eeff4" Oct 29 23:15:20.211086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.210695 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm" event={"ID":"2b421f85-4894-47f3-9e87-c5b683ea4e6b","Type":"ContainerDied","Data":"67fa5e42d1e1865e6ed36d37ddeb10b3d7b8d214cb7da8c3ea43d2e6829575e8"} Oct 29 23:15:20.211342 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.211320 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm" Oct 29 23:15:20.217284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.217193 4821 generic.go:356] "Generic (PLEG): container finished" podID="c0219f30-2070-4cf8-bdfa-304ad5c86ad7" containerID="dd6b8d7c5081a3b582abfb2b3c62211305bee5c03bd6e71eafc2c3da1fe621cd" exitCode=60 Oct 29 23:15:20.217398 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.217319 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5" event={"ID":"c0219f30-2070-4cf8-bdfa-304ad5c86ad7","Type":"ContainerDied","Data":"dd6b8d7c5081a3b582abfb2b3c62211305bee5c03bd6e71eafc2c3da1fe621cd"} Oct 29 23:15:20.219696 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.219671 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97" Oct 29 23:15:20.220468 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.219842 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97" event={"ID":"514b5faa-73ee-4d54-9e6b-c474a52d82ca","Type":"ContainerDied","Data":"5482b8e512f66652e09a544c6b807b6c7ccb6517153db9594916bd57b65aa12d"} Oct 29 23:15:20.223009 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.222986 4821 generic.go:356] "Generic (PLEG): container finished" podID="786f75a2-8350-4081-8ca4-041ed3247252" containerID="15057460d6abe1583fc1458c6cf762505f6550ed82b3de7c483516c859535c3b" exitCode=60 Oct 29 23:15:20.223538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.223517 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf" event={"ID":"786f75a2-8350-4081-8ca4-041ed3247252","Type":"ContainerDied","Data":"15057460d6abe1583fc1458c6cf762505f6550ed82b3de7c483516c859535c3b"} Oct 29 23:15:20.230054 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.229982 4821 scope.go:122] "RemoveContainer" containerID="dd6b8d7c5081a3b582abfb2b3c62211305bee5c03bd6e71eafc2c3da1fe621cd" Oct 29 23:15:20.230755 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:20.230732 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5_odh-ai-gateway-infra(c0219f30-2070-4cf8-bdfa-304ad5c86ad7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5" podUID="c0219f30-2070-4cf8-bdfa-304ad5c86ad7" Oct 29 23:15:20.231832 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.231797 4821 scope.go:122] "RemoveContainer" containerID="15057460d6abe1583fc1458c6cf762505f6550ed82b3de7c483516c859535c3b" Oct 29 23:15:20.232246 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:20.232218 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf_odh-ai-gateway-infra(786f75a2-8350-4081-8ca4-041ed3247252)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf" podUID="786f75a2-8350-4081-8ca4-041ed3247252" Oct 29 23:15:20.242782 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.242749 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g" event={"ID":"17e44a4a-8eb9-4732-8924-d76683d1e213","Type":"ContainerDied","Data":"f3bb72277fb9d0aca2fdf67f2402733e98aa59158bab897cc4af6ee1a3a76824"} Oct 29 23:15:20.242939 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.242886 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g" Oct 29 23:15:20.253358 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.253334 4821 scope.go:122] "RemoveContainer" containerID="ba980c5a021f4316289ecc6a44b8787d50ae7ae72103619dce324888870c801e" Oct 29 23:15:20.277389 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.277360 4821 scope.go:122] "RemoveContainer" containerID="2068bd6ab12c5743bfcbee87a3bd07029f7abc733a738e0d2becfa9023e28f73" Oct 29 23:15:20.296803 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.296779 4821 scope.go:122] "RemoveContainer" containerID="f7ee647f314e0168543f69e92698bd03781e58630ce228c136d13c6fc2a62986" Oct 29 23:15:20.307784 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.307753 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf"] Oct 29 23:15:20.310454 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.310429 4821 scope.go:122] "RemoveContainer" containerID="d676204ec9fe9ff1fd07988c60a6e267ade773c07960db5cfd54ffd03ac434a1" Oct 29 23:15:20.347265 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.347235 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp"] Oct 29 23:15:20.354935 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.354218 4821 scope.go:122] "RemoveContainer" containerID="0985436bd6ced5dba5a38f6ff7b5dc6c20b0cf7090515b0fc5e79ad3c180ddf4" Oct 29 23:15:20.455850 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.455823 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6" Oct 29 23:15:20.488462 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.488427 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm"] Oct 29 23:15:20.498433 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.498397 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-3e5b21-29888595-cgllm"] Oct 29 23:15:20.503212 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.503166 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5"] Oct 29 23:15:20.538633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.538603 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g"] Oct 29 23:15:20.546484 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.546458 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51189101-afc0-42dd-815e-a8330ec8b42a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51189101-afc0-42dd-815e-a8330ec8b42a-maas-api-service-ca\") pod \"51189101-afc0-42dd-815e-a8330ec8b42a\" (UID: \"51189101-afc0-42dd-815e-a8330ec8b42a\") " Oct 29 23:15:20.546605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.546531 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51189101-afc0-42dd-815e-a8330ec8b42a-kube-api-access-fk5hn\" (UniqueName: \"kubernetes.io/projected/51189101-afc0-42dd-815e-a8330ec8b42a-kube-api-access-fk5hn\") pod \"51189101-afc0-42dd-815e-a8330ec8b42a\" (UID: \"51189101-afc0-42dd-815e-a8330ec8b42a\") " Oct 29 23:15:20.546948 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.546912 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51189101-afc0-42dd-815e-a8330ec8b42a-maas-api-service-ca" pod "51189101-afc0-42dd-815e-a8330ec8b42a" (UID: "51189101-afc0-42dd-815e-a8330ec8b42a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:20.547190 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.547175 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51189101-afc0-42dd-815e-a8330ec8b42a-maas-api-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:20.548588 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.548567 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-67363b-29888595-4v82g"] Oct 29 23:15:20.549017 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.548990 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51189101-afc0-42dd-815e-a8330ec8b42a-kube-api-access-fk5hn" pod "51189101-afc0-42dd-815e-a8330ec8b42a" (UID: "51189101-afc0-42dd-815e-a8330ec8b42a"). InnerVolumeSpecName "kube-api-access-fk5hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:20.584458 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.584320 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97"] Oct 29 23:15:20.590425 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.590401 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b0c3c3-29888595-f8h97"] Oct 29 23:15:20.649587 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:20.649555 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk5hn\" (UniqueName: \"kubernetes.io/projected/51189101-afc0-42dd-815e-a8330ec8b42a-kube-api-access-fk5hn\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:21.259081 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.259056 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6" Oct 29 23:15:21.259301 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.259049 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6" event={"ID":"51189101-afc0-42dd-815e-a8330ec8b42a","Type":"ContainerDied","Data":"1eaa2234227e7659903c647ed63048cb30d0d0d9840e4915b4f83638387e68fd"} Oct 29 23:15:21.259301 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.259213 4821 scope.go:122] "RemoveContainer" containerID="00047ce3ca85fdd758875f264d75d5c1ffccca3fce2571503d7389f93c8c5f83" Oct 29 23:15:21.263386 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.263364 4821 generic.go:356] "Generic (PLEG): container finished" podID="7b4b604b-3757-4c03-a745-f01e02ce82c2" containerID="82511d2118b3ca34301c78de7cac49a8af17c71f239b6081681bb92684a01b94" exitCode=60 Oct 29 23:15:21.263507 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.263476 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8" event={"ID":"7b4b604b-3757-4c03-a745-f01e02ce82c2","Type":"ContainerDied","Data":"82511d2118b3ca34301c78de7cac49a8af17c71f239b6081681bb92684a01b94"} Oct 29 23:15:21.263996 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.263975 4821 scope.go:122] "RemoveContainer" containerID="82511d2118b3ca34301c78de7cac49a8af17c71f239b6081681bb92684a01b94" Oct 29 23:15:21.264352 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:21.264327 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8_odh-ai-gateway-infra(7b4b604b-3757-4c03-a745-f01e02ce82c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8" podUID="7b4b604b-3757-4c03-a745-f01e02ce82c2" Oct 29 23:15:21.294469 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.293813 4821 scope.go:122] "RemoveContainer" containerID="a6e9a8ceb935a0091a00ba9ab03fa67799bc2509bfb0a3038ec59a6cc908580f" Oct 29 23:15:21.501699 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.501643 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6"] Oct 29 23:15:21.518215 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.514562 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-08e15d-29888595-96tw6"] Oct 29 23:15:21.526388 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.526361 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf" Oct 29 23:15:21.566102 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.566076 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5" Oct 29 23:15:21.571116 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.571090 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp" Oct 29 23:15:21.665897 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.665858 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0219f30-2070-4cf8-bdfa-304ad5c86ad7-kube-api-access-8f8v7\" (UniqueName: \"kubernetes.io/projected/c0219f30-2070-4cf8-bdfa-304ad5c86ad7-kube-api-access-8f8v7\") pod \"c0219f30-2070-4cf8-bdfa-304ad5c86ad7\" (UID: \"c0219f30-2070-4cf8-bdfa-304ad5c86ad7\") " Oct 29 23:15:21.666102 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.665947 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/786f75a2-8350-4081-8ca4-041ed3247252-kube-api-access-5bc6m\" (UniqueName: \"kubernetes.io/projected/786f75a2-8350-4081-8ca4-041ed3247252-kube-api-access-5bc6m\") pod \"786f75a2-8350-4081-8ca4-041ed3247252\" (UID: \"786f75a2-8350-4081-8ca4-041ed3247252\") " Oct 29 23:15:21.666102 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.666006 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff81e511-3cab-4008-b2a5-1ab9774eeff4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff81e511-3cab-4008-b2a5-1ab9774eeff4-maas-api-service-ca\") pod \"ff81e511-3cab-4008-b2a5-1ab9774eeff4\" (UID: \"ff81e511-3cab-4008-b2a5-1ab9774eeff4\") " Oct 29 23:15:21.666271 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.666188 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff81e511-3cab-4008-b2a5-1ab9774eeff4-kube-api-access-h5jtr\" (UniqueName: \"kubernetes.io/projected/ff81e511-3cab-4008-b2a5-1ab9774eeff4-kube-api-access-h5jtr\") pod \"ff81e511-3cab-4008-b2a5-1ab9774eeff4\" (UID: \"ff81e511-3cab-4008-b2a5-1ab9774eeff4\") " Oct 29 23:15:21.666271 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.666239 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0219f30-2070-4cf8-bdfa-304ad5c86ad7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0219f30-2070-4cf8-bdfa-304ad5c86ad7-maas-api-service-ca\") pod \"c0219f30-2070-4cf8-bdfa-304ad5c86ad7\" (UID: \"c0219f30-2070-4cf8-bdfa-304ad5c86ad7\") " Oct 29 23:15:21.666395 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.666295 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/786f75a2-8350-4081-8ca4-041ed3247252-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786f75a2-8350-4081-8ca4-041ed3247252-maas-api-service-ca\") pod \"786f75a2-8350-4081-8ca4-041ed3247252\" (UID: \"786f75a2-8350-4081-8ca4-041ed3247252\") " Oct 29 23:15:21.667337 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.667288 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff81e511-3cab-4008-b2a5-1ab9774eeff4-maas-api-service-ca" pod "ff81e511-3cab-4008-b2a5-1ab9774eeff4" (UID: "ff81e511-3cab-4008-b2a5-1ab9774eeff4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:21.667561 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.667329 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0219f30-2070-4cf8-bdfa-304ad5c86ad7-maas-api-service-ca" pod "c0219f30-2070-4cf8-bdfa-304ad5c86ad7" (UID: "c0219f30-2070-4cf8-bdfa-304ad5c86ad7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:21.667649 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.667587 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/786f75a2-8350-4081-8ca4-041ed3247252-maas-api-service-ca" pod "786f75a2-8350-4081-8ca4-041ed3247252" (UID: "786f75a2-8350-4081-8ca4-041ed3247252"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:21.670362 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.670339 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff81e511-3cab-4008-b2a5-1ab9774eeff4-kube-api-access-h5jtr" pod "ff81e511-3cab-4008-b2a5-1ab9774eeff4" (UID: "ff81e511-3cab-4008-b2a5-1ab9774eeff4"). InnerVolumeSpecName "kube-api-access-h5jtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:21.670716 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.670684 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/786f75a2-8350-4081-8ca4-041ed3247252-kube-api-access-5bc6m" pod "786f75a2-8350-4081-8ca4-041ed3247252" (UID: "786f75a2-8350-4081-8ca4-041ed3247252"). InnerVolumeSpecName "kube-api-access-5bc6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:21.672292 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.672254 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0219f30-2070-4cf8-bdfa-304ad5c86ad7-kube-api-access-8f8v7" pod "c0219f30-2070-4cf8-bdfa-304ad5c86ad7" (UID: "c0219f30-2070-4cf8-bdfa-304ad5c86ad7"). InnerVolumeSpecName "kube-api-access-8f8v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:21.770403 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.769498 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5jtr\" (UniqueName: \"kubernetes.io/projected/ff81e511-3cab-4008-b2a5-1ab9774eeff4-kube-api-access-h5jtr\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:21.770403 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.769538 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0219f30-2070-4cf8-bdfa-304ad5c86ad7-maas-api-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:21.770403 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.769553 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786f75a2-8350-4081-8ca4-041ed3247252-maas-api-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:21.770403 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.769568 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f8v7\" (UniqueName: \"kubernetes.io/projected/c0219f30-2070-4cf8-bdfa-304ad5c86ad7-kube-api-access-8f8v7\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:21.770403 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.769582 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bc6m\" (UniqueName: \"kubernetes.io/projected/786f75a2-8350-4081-8ca4-041ed3247252-kube-api-access-5bc6m\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:21.770403 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:21.769594 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff81e511-3cab-4008-b2a5-1ab9774eeff4-maas-api-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:22.192225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.189054 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17e44a4a-8eb9-4732-8924-d76683d1e213" path="/var/lib/kubelet/pods/17e44a4a-8eb9-4732-8924-d76683d1e213/volumes" Oct 29 23:15:22.192225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.189867 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b421f85-4894-47f3-9e87-c5b683ea4e6b" path="/var/lib/kubelet/pods/2b421f85-4894-47f3-9e87-c5b683ea4e6b/volumes" Oct 29 23:15:22.192225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.190492 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51189101-afc0-42dd-815e-a8330ec8b42a" path="/var/lib/kubelet/pods/51189101-afc0-42dd-815e-a8330ec8b42a/volumes" Oct 29 23:15:22.192225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.191503 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="514b5faa-73ee-4d54-9e6b-c474a52d82ca" path="/var/lib/kubelet/pods/514b5faa-73ee-4d54-9e6b-c474a52d82ca/volumes" Oct 29 23:15:22.282363 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.282332 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf" Oct 29 23:15:22.282542 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.282335 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf" event={"ID":"786f75a2-8350-4081-8ca4-041ed3247252","Type":"ContainerDied","Data":"15ddd722c083028be0854a0e13ebc6c388d0584cabbbb8ea25485fdb9d1bdb4c"} Oct 29 23:15:22.282542 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.282483 4821 scope.go:122] "RemoveContainer" containerID="15057460d6abe1583fc1458c6cf762505f6550ed82b3de7c483516c859535c3b" Oct 29 23:15:22.293527 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.293487 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp" event={"ID":"ff81e511-3cab-4008-b2a5-1ab9774eeff4","Type":"ContainerDied","Data":"3000b6901097762b061cb02a759c022076211042a1cabd99ea0a4811f6f26c38"} Oct 29 23:15:22.293713 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.293624 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp" Oct 29 23:15:22.300118 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.300080 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5" event={"ID":"c0219f30-2070-4cf8-bdfa-304ad5c86ad7","Type":"ContainerDied","Data":"e3bbb6509c0b73acf60504bfb9404a63854d4b24a5a6a030eec22f71b9011631"} Oct 29 23:15:22.300308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.300212 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5" Oct 29 23:15:22.305908 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.305885 4821 scope.go:122] "RemoveContainer" containerID="0967ff9f9dba158188845b788afa3558b174e2d7a1f8231cb31fed0683fff04c" Oct 29 23:15:22.327304 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.327147 4821 scope.go:122] "RemoveContainer" containerID="dd6b8d7c5081a3b582abfb2b3c62211305bee5c03bd6e71eafc2c3da1fe621cd" Oct 29 23:15:22.328027 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.327899 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4"] Oct 29 23:15:22.359983 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.359489 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf"] Oct 29 23:15:22.366274 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.366244 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b2921b-29888595-bxstf"] Oct 29 23:15:22.371174 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.371148 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8"] Oct 29 23:15:22.402965 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.402930 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp"] Oct 29 23:15:22.408421 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.408390 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-755453-29888595-m8ljp"] Oct 29 23:15:22.594318 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.590658 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5"] Oct 29 23:15:22.599233 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.597447 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-49d87f-29888595-jv4r5"] Oct 29 23:15:22.824012 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.823981 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8" Oct 29 23:15:22.914339 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.914310 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4b604b-3757-4c03-a745-f01e02ce82c2-kube-api-access-jwmkq\" (UniqueName: \"kubernetes.io/projected/7b4b604b-3757-4c03-a745-f01e02ce82c2-kube-api-access-jwmkq\") pod \"7b4b604b-3757-4c03-a745-f01e02ce82c2\" (UID: \"7b4b604b-3757-4c03-a745-f01e02ce82c2\") " Oct 29 23:15:22.914530 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.914504 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b4b604b-3757-4c03-a745-f01e02ce82c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b4b604b-3757-4c03-a745-f01e02ce82c2-maas-api-service-ca\") pod \"7b4b604b-3757-4c03-a745-f01e02ce82c2\" (UID: \"7b4b604b-3757-4c03-a745-f01e02ce82c2\") " Oct 29 23:15:22.914925 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.914891 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b4b604b-3757-4c03-a745-f01e02ce82c2-maas-api-service-ca" pod "7b4b604b-3757-4c03-a745-f01e02ce82c2" (UID: "7b4b604b-3757-4c03-a745-f01e02ce82c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:22.915693 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.915666 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b4b604b-3757-4c03-a745-f01e02ce82c2-maas-api-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:22.916992 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:22.916963 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4b604b-3757-4c03-a745-f01e02ce82c2-kube-api-access-jwmkq" pod "7b4b604b-3757-4c03-a745-f01e02ce82c2" (UID: "7b4b604b-3757-4c03-a745-f01e02ce82c2"). InnerVolumeSpecName "kube-api-access-jwmkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:23.017150 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:23.017116 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwmkq\" (UniqueName: \"kubernetes.io/projected/7b4b604b-3757-4c03-a745-f01e02ce82c2-kube-api-access-jwmkq\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:23.447223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:23.445380 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8" event={"ID":"7b4b604b-3757-4c03-a745-f01e02ce82c2","Type":"ContainerDied","Data":"43f8d273d19c2e7a29e6563ea0407cabaae6f03abb0fd03e28b54e742d50dbcb"} Oct 29 23:15:23.447223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:23.445460 4821 scope.go:122] "RemoveContainer" containerID="82511d2118b3ca34301c78de7cac49a8af17c71f239b6081681bb92684a01b94" Oct 29 23:15:23.447223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:23.445586 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8" Oct 29 23:15:23.611517 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:23.611477 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8"] Oct 29 23:15:23.617160 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:23.617129 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-17394a-29888595-l2sf8"] Oct 29 23:15:23.897080 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:23.896996 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:15:24.177276 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:24.177131 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="786f75a2-8350-4081-8ca4-041ed3247252" path="/var/lib/kubelet/pods/786f75a2-8350-4081-8ca4-041ed3247252/volumes" Oct 29 23:15:24.177940 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:24.177915 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b4b604b-3757-4c03-a745-f01e02ce82c2" path="/var/lib/kubelet/pods/7b4b604b-3757-4c03-a745-f01e02ce82c2/volumes" Oct 29 23:15:24.178625 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:24.178587 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0219f30-2070-4cf8-bdfa-304ad5c86ad7" path="/var/lib/kubelet/pods/c0219f30-2070-4cf8-bdfa-304ad5c86ad7/volumes" Oct 29 23:15:24.180563 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:24.179647 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff81e511-3cab-4008-b2a5-1ab9774eeff4" path="/var/lib/kubelet/pods/ff81e511-3cab-4008-b2a5-1ab9774eeff4/volumes" Oct 29 23:15:29.543230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:29.522299 4821 scope.go:122] "RemoveContainer" containerID="a215e3d0cde79944ffa5357b90a0c81a494071d8d60fa9b54f3fb237b08411dc" Oct 29 23:15:29.594402 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:29.594379 4821 scope.go:122] "RemoveContainer" containerID="d3474112506867ad785d0bd631ff6ea3d0ef2083402ddfb6bc9273b7d64b0165" Oct 29 23:15:29.609837 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:29.609811 4821 scope.go:122] "RemoveContainer" containerID="1871320a06307a5e8df7fdd9dd5bddc3e9ee3540ae7dc23925b39a3fe06e2c10" Oct 29 23:15:29.649788 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:29.631347 4821 scope.go:122] "RemoveContainer" containerID="afd932e4286570989947c901d58b5dd981d6239d6bb6f4aba380f172ea78732c" Oct 29 23:15:37.367591 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:37.367549 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" podUID="6a19fff5-f4be-494c-97ad-3520546346d1" containerName="main" containerID="cri-o://9cce07ab2eb3fb071565ff3268fa6e8adb951ebee99c3a3ead636a415cf2676d" gracePeriod=45 Oct 29 23:15:37.426042 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:37.426003 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" podUID="6a19fff5-f4be-494c-97ad-3520546346d1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.208:8000/ready\": dial tcp 10.133.0.208:8000: connect: connection refused" Oct 29 23:15:37.887852 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:37.887822 4821 generic.go:356] "Generic (PLEG): container finished" podID="6a19fff5-f4be-494c-97ad-3520546346d1" containerID="9cce07ab2eb3fb071565ff3268fa6e8adb951ebee99c3a3ead636a415cf2676d" exitCode=0 Oct 29 23:15:37.888024 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:37.887897 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" event={"ID":"6a19fff5-f4be-494c-97ad-3520546346d1","Type":"ContainerDied","Data":"9cce07ab2eb3fb071565ff3268fa6e8adb951ebee99c3a3ead636a415cf2676d"} Oct 29 23:15:37.888024 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:37.887937 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" event={"ID":"6a19fff5-f4be-494c-97ad-3520546346d1","Type":"ContainerDied","Data":"2c3ea3f232af994abc4411a50a41ae5488e2c279a5b72b8a92e154419c68d999"} Oct 29 23:15:37.888024 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:37.887948 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c3ea3f232af994abc4411a50a41ae5488e2c279a5b72b8a92e154419c68d999" Oct 29 23:15:37.976832 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:37.976756 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:15:38.134649 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:38.134616 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-tmp-dir\") pod \"6a19fff5-f4be-494c-97ad-3520546346d1\" (UID: \"6a19fff5-f4be-494c-97ad-3520546346d1\") " Oct 29 23:15:38.134854 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:38.134700 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-home\" (UniqueName: \"kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-home\") pod \"6a19fff5-f4be-494c-97ad-3520546346d1\" (UID: \"6a19fff5-f4be-494c-97ad-3520546346d1\") " Oct 29 23:15:38.134854 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:38.134745 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a19fff5-f4be-494c-97ad-3520546346d1-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a19fff5-f4be-494c-97ad-3520546346d1-tls-certs\") pod \"6a19fff5-f4be-494c-97ad-3520546346d1\" (UID: \"6a19fff5-f4be-494c-97ad-3520546346d1\") " Oct 29 23:15:38.134993 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:38.134856 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-model-cache\") pod \"6a19fff5-f4be-494c-97ad-3520546346d1\" (UID: \"6a19fff5-f4be-494c-97ad-3520546346d1\") " Oct 29 23:15:38.134993 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:38.134914 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-dshm\") pod \"6a19fff5-f4be-494c-97ad-3520546346d1\" (UID: \"6a19fff5-f4be-494c-97ad-3520546346d1\") " Oct 29 23:15:38.134993 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:38.134927 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-tmp-dir" pod "6a19fff5-f4be-494c-97ad-3520546346d1" (UID: "6a19fff5-f4be-494c-97ad-3520546346d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:38.135141 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:38.135004 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-home" pod "6a19fff5-f4be-494c-97ad-3520546346d1" (UID: "6a19fff5-f4be-494c-97ad-3520546346d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:38.135141 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:38.135098 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a19fff5-f4be-494c-97ad-3520546346d1-kube-api-access-ch8t6\" (UniqueName: \"kubernetes.io/projected/6a19fff5-f4be-494c-97ad-3520546346d1-kube-api-access-ch8t6\") pod \"6a19fff5-f4be-494c-97ad-3520546346d1\" (UID: \"6a19fff5-f4be-494c-97ad-3520546346d1\") " Oct 29 23:15:38.135282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:38.135139 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-model-cache" pod "6a19fff5-f4be-494c-97ad-3520546346d1" (UID: "6a19fff5-f4be-494c-97ad-3520546346d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:38.135946 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:38.135924 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:38.135946 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:38.135944 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:38.136165 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:38.135953 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:38.137393 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:38.137368 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a19fff5-f4be-494c-97ad-3520546346d1-tls-certs" pod "6a19fff5-f4be-494c-97ad-3520546346d1" (UID: "6a19fff5-f4be-494c-97ad-3520546346d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:15:38.137544 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:38.137527 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a19fff5-f4be-494c-97ad-3520546346d1-kube-api-access-ch8t6" pod "6a19fff5-f4be-494c-97ad-3520546346d1" (UID: "6a19fff5-f4be-494c-97ad-3520546346d1"). InnerVolumeSpecName "kube-api-access-ch8t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:38.137601 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:38.137590 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-dshm" pod "6a19fff5-f4be-494c-97ad-3520546346d1" (UID: "6a19fff5-f4be-494c-97ad-3520546346d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:38.237189 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:38.237113 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch8t6\" (UniqueName: \"kubernetes.io/projected/6a19fff5-f4be-494c-97ad-3520546346d1-kube-api-access-ch8t6\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:38.237189 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:38.237148 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a19fff5-f4be-494c-97ad-3520546346d1-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:38.237189 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:38.237159 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a19fff5-f4be-494c-97ad-3520546346d1-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:38.896391 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:38.896359 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4" Oct 29 23:15:38.954793 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:38.954762 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4"] Oct 29 23:15:38.967819 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:38.967788 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-67363b/auth-test-model-67363b-kserve-5df8986f7-5zdc4"] Oct 29 23:15:39.803436 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:39.803357 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk"] Oct 29 23:15:39.821520 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:39.803564 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" podUID="2f71bdab-b1a2-4513-a7f1-2455e4222efa" containerName="istio-proxy" containerID="cri-o://b2306d76e7c3490ed16fb67412a842b2a467b0b846a0334bc06bb0bcd4835090" gracePeriod=30 Oct 29 23:15:40.171612 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:40.171577 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a19fff5-f4be-494c-97ad-3520546346d1" path="/var/lib/kubelet/pods/6a19fff5-f4be-494c-97ad-3520546346d1/volumes" Oct 29 23:15:40.918771 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:40.918742 4821 generic.go:356] "Generic (PLEG): container finished" podID="f275822e-4f33-4366-8be5-348dc70ebffd" containerID="19e90a2a30dd1b2974e6f079979fea2d04f67623ea7fe4eafb908ebb571244b9" exitCode=0 Oct 29 23:15:40.918950 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:40.918791 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" event={"ID":"f275822e-4f33-4366-8be5-348dc70ebffd","Type":"ContainerDied","Data":"19e90a2a30dd1b2974e6f079979fea2d04f67623ea7fe4eafb908ebb571244b9"} Oct 29 23:15:40.919309 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:40.919289 4821 scope.go:122] "RemoveContainer" containerID="19e90a2a30dd1b2974e6f079979fea2d04f67623ea7fe4eafb908ebb571244b9" Oct 29 23:15:40.921195 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:40.921168 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:15:40.921303 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:40.921263 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:15:40.921303 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:40.921297 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:15:41.529355 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:41.529323 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:15:41.967404 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:41.967364 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" event={"ID":"f275822e-4f33-4366-8be5-348dc70ebffd","Type":"ContainerStarted","Data":"ad7ff2d1b41691e6031661823937c2475751e52c32a59610ad4416e4b781d647"} Oct 29 23:15:42.205028 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:42.204993 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:15:42.206251 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:42.206217 4821 patch_prober.go:29] interesting pod/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body= Oct 29 23:15:42.206402 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:42.206280 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" podUID="f275822e-4f33-4366-8be5-348dc70ebffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" Oct 29 23:15:43.205014 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:43.204970 4821 patch_prober.go:29] interesting pod/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body= Oct 29 23:15:43.205563 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:43.205041 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" podUID="f275822e-4f33-4366-8be5-348dc70ebffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" Oct 29 23:15:44.205553 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:44.205516 4821 patch_prober.go:29] interesting pod/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body= Oct 29 23:15:44.206057 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:44.206027 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" podUID="f275822e-4f33-4366-8be5-348dc70ebffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" Oct 29 23:15:45.002237 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.002175 4821 generic.go:356] "Generic (PLEG): container finished" podID="2f71bdab-b1a2-4513-a7f1-2455e4222efa" containerID="b2306d76e7c3490ed16fb67412a842b2a467b0b846a0334bc06bb0bcd4835090" exitCode=0 Oct 29 23:15:45.002435 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.002258 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" event={"ID":"2f71bdab-b1a2-4513-a7f1-2455e4222efa","Type":"ContainerDied","Data":"b2306d76e7c3490ed16fb67412a842b2a467b0b846a0334bc06bb0bcd4835090"} Oct 29 23:15:45.181497 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.181471 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:15:45.204913 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.204821 4821 patch_prober.go:29] interesting pod/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body= Oct 29 23:15:45.204913 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.204895 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" podUID="f275822e-4f33-4366-8be5-348dc70ebffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" Oct 29 23:15:45.291461 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.291419 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f71bdab-b1a2-4513-a7f1-2455e4222efa-kube-api-access-2vgxq\" (UniqueName: \"kubernetes.io/projected/2f71bdab-b1a2-4513-a7f1-2455e4222efa-kube-api-access-2vgxq\") pod \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " Oct 29 23:15:45.291951 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.291592 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-workload-certs\") pod \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " Oct 29 23:15:45.291951 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.291685 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istiod-ca-cert\") pod \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " Oct 29 23:15:45.291951 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.291755 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-token\" (UniqueName: \"kubernetes.io/projected/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-token\") pod \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " Oct 29 23:15:45.291951 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.291821 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-data\") pod \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " Oct 29 23:15:45.291951 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.291876 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-podinfo\") pod \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " Oct 29 23:15:45.291951 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.291901 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-workload-certs" pod "2f71bdab-b1a2-4513-a7f1-2455e4222efa" (UID: "2f71bdab-b1a2-4513-a7f1-2455e4222efa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:45.291951 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.291909 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-credential-socket\") pod \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " Oct 29 23:15:45.292456 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.292042 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istiod-ca-cert" pod "2f71bdab-b1a2-4513-a7f1-2455e4222efa" (UID: "2f71bdab-b1a2-4513-a7f1-2455e4222efa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:15:45.292456 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.292131 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-credential-socket" pod "2f71bdab-b1a2-4513-a7f1-2455e4222efa" (UID: "2f71bdab-b1a2-4513-a7f1-2455e4222efa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:45.292456 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.292080 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-workload-socket\") pod \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " Oct 29 23:15:45.292456 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.292256 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-workload-socket" pod "2f71bdab-b1a2-4513-a7f1-2455e4222efa" (UID: "2f71bdab-b1a2-4513-a7f1-2455e4222efa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:45.292456 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.292272 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-envoy\") pod \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\" (UID: \"2f71bdab-b1a2-4513-a7f1-2455e4222efa\") " Oct 29 23:15:45.292456 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.292340 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-data" pod "2f71bdab-b1a2-4513-a7f1-2455e4222efa" (UID: "2f71bdab-b1a2-4513-a7f1-2455e4222efa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:45.293370 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.293347 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:45.293601 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.293581 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:45.293760 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.293745 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:45.293863 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.293852 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:45.293952 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.293941 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:45.294816 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.294790 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-podinfo" pod "2f71bdab-b1a2-4513-a7f1-2455e4222efa" (UID: "2f71bdab-b1a2-4513-a7f1-2455e4222efa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:15:45.295309 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.295283 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-token" pod "2f71bdab-b1a2-4513-a7f1-2455e4222efa" (UID: "2f71bdab-b1a2-4513-a7f1-2455e4222efa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:45.295309 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.295296 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-envoy" pod "2f71bdab-b1a2-4513-a7f1-2455e4222efa" (UID: "2f71bdab-b1a2-4513-a7f1-2455e4222efa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:15:45.295629 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.295595 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f71bdab-b1a2-4513-a7f1-2455e4222efa-kube-api-access-2vgxq" pod "2f71bdab-b1a2-4513-a7f1-2455e4222efa" (UID: "2f71bdab-b1a2-4513-a7f1-2455e4222efa"). InnerVolumeSpecName "kube-api-access-2vgxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:15:45.397135 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.396972 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:45.397135 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.397010 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vgxq\" (UniqueName: \"kubernetes.io/projected/2f71bdab-b1a2-4513-a7f1-2455e4222efa-kube-api-access-2vgxq\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:45.397135 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.397026 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:45.397135 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:45.397039 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f71bdab-b1a2-4513-a7f1-2455e4222efa-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:15:46.032227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:46.029643 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" event={"ID":"2f71bdab-b1a2-4513-a7f1-2455e4222efa","Type":"ContainerDied","Data":"11f05b804161ef06fc0b6e3f061bb8bcfae92710818edf840af7a99f24bb124d"} Oct 29 23:15:46.032227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:46.029779 4821 scope.go:122] "RemoveContainer" containerID="b2306d76e7c3490ed16fb67412a842b2a467b0b846a0334bc06bb0bcd4835090" Oct 29 23:15:46.032227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:46.030082 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk" Oct 29 23:15:46.145708 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:46.145669 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk"] Oct 29 23:15:46.150157 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:46.150131 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-r6wmk"] Oct 29 23:15:46.172673 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:46.172645 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f71bdab-b1a2-4513-a7f1-2455e4222efa" path="/var/lib/kubelet/pods/2f71bdab-b1a2-4513-a7f1-2455e4222efa/volumes" Oct 29 23:15:46.205731 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:46.205693 4821 patch_prober.go:29] interesting pod/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body= Oct 29 23:15:46.205864 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:46.205755 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" podUID="f275822e-4f33-4366-8be5-348dc70ebffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" Oct 29 23:15:47.205958 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:47.205918 4821 patch_prober.go:29] interesting pod/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body= Oct 29 23:15:47.206660 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:47.206627 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" podUID="f275822e-4f33-4366-8be5-348dc70ebffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" Oct 29 23:15:48.209763 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.209733 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:15:48.210242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.210224 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:15:48.210863 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.210845 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:15:48.702574 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.702541 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh"] Oct 29 23:15:48.703242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703182 4821 cpu_manager.go:414] "containerMap: removing container" podUID="7b4b604b-3757-4c03-a745-f01e02ce82c2" containerName="cleanup" Oct 29 23:15:48.703242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703218 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4b604b-3757-4c03-a745-f01e02ce82c2" containerName="cleanup" Oct 29 23:15:48.703242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703237 4821 cpu_manager.go:414] "containerMap: removing container" podUID="6a19fff5-f4be-494c-97ad-3520546346d1" containerName="main" Oct 29 23:15:48.703242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703244 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a19fff5-f4be-494c-97ad-3520546346d1" containerName="main" Oct 29 23:15:48.703428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703254 4821 cpu_manager.go:414] "containerMap: removing container" podUID="2f71bdab-b1a2-4513-a7f1-2455e4222efa" containerName="istio-proxy" Oct 29 23:15:48.703428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703260 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f71bdab-b1a2-4513-a7f1-2455e4222efa" containerName="istio-proxy" Oct 29 23:15:48.703428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703271 4821 cpu_manager.go:414] "containerMap: removing container" podUID="17e44a4a-8eb9-4732-8924-d76683d1e213" containerName="cleanup" Oct 29 23:15:48.703428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703276 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e44a4a-8eb9-4732-8924-d76683d1e213" containerName="cleanup" Oct 29 23:15:48.703428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703284 4821 cpu_manager.go:414] "containerMap: removing container" podUID="c0da6fd9-efd5-4587-b764-524010b6a352" containerName="istio-proxy" Oct 29 23:15:48.703428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703291 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0da6fd9-efd5-4587-b764-524010b6a352" containerName="istio-proxy" Oct 29 23:15:48.703428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703320 4821 cpu_manager.go:414] "containerMap: removing container" podUID="2b421f85-4894-47f3-9e87-c5b683ea4e6b" containerName="cleanup" Oct 29 23:15:48.703428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703327 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b421f85-4894-47f3-9e87-c5b683ea4e6b" containerName="cleanup" Oct 29 23:15:48.703428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703335 4821 cpu_manager.go:414] "containerMap: removing container" podUID="ff81e511-3cab-4008-b2a5-1ab9774eeff4" containerName="cleanup" Oct 29 23:15:48.703428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703343 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff81e511-3cab-4008-b2a5-1ab9774eeff4" containerName="cleanup" Oct 29 23:15:48.703428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703353 4821 cpu_manager.go:414] "containerMap: removing container" podUID="137c7cf5-e922-49d1-acbc-fc0b13899da5" containerName="istio-proxy" Oct 29 23:15:48.703428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703360 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137c7cf5-e922-49d1-acbc-fc0b13899da5" containerName="istio-proxy" Oct 29 23:15:48.703428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703369 4821 cpu_manager.go:414] "containerMap: removing container" podUID="17e44a4a-8eb9-4732-8924-d76683d1e213" containerName="cleanup" Oct 29 23:15:48.703428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703375 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e44a4a-8eb9-4732-8924-d76683d1e213" containerName="cleanup" Oct 29 23:15:48.703428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703388 4821 cpu_manager.go:414] "containerMap: removing container" podUID="ff81e511-3cab-4008-b2a5-1ab9774eeff4" containerName="cleanup" Oct 29 23:15:48.703428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703394 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff81e511-3cab-4008-b2a5-1ab9774eeff4" containerName="cleanup" Oct 29 23:15:48.703428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703413 4821 cpu_manager.go:414] "containerMap: removing container" podUID="786f75a2-8350-4081-8ca4-041ed3247252" containerName="cleanup" Oct 29 23:15:48.703428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703420 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="786f75a2-8350-4081-8ca4-041ed3247252" containerName="cleanup" Oct 29 23:15:48.703428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703429 4821 cpu_manager.go:414] "containerMap: removing container" podUID="74bc6f99-ab50-4bd6-9b61-a8727d36f879" containerName="revoke-keys" Oct 29 23:15:48.703428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703435 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74bc6f99-ab50-4bd6-9b61-a8727d36f879" containerName="revoke-keys" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703444 4821 cpu_manager.go:414] "containerMap: removing container" podUID="514b5faa-73ee-4d54-9e6b-c474a52d82ca" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703450 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="514b5faa-73ee-4d54-9e6b-c474a52d82ca" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703493 4821 cpu_manager.go:414] "containerMap: removing container" podUID="51189101-afc0-42dd-815e-a8330ec8b42a" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703500 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51189101-afc0-42dd-815e-a8330ec8b42a" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703508 4821 cpu_manager.go:414] "containerMap: removing container" podUID="786f75a2-8350-4081-8ca4-041ed3247252" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703515 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="786f75a2-8350-4081-8ca4-041ed3247252" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703535 4821 cpu_manager.go:414] "containerMap: removing container" podUID="786f75a2-8350-4081-8ca4-041ed3247252" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703542 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="786f75a2-8350-4081-8ca4-041ed3247252" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703556 4821 cpu_manager.go:414] "containerMap: removing container" podUID="514b5faa-73ee-4d54-9e6b-c474a52d82ca" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703562 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="514b5faa-73ee-4d54-9e6b-c474a52d82ca" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703577 4821 cpu_manager.go:414] "containerMap: removing container" podUID="c0219f30-2070-4cf8-bdfa-304ad5c86ad7" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703583 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0219f30-2070-4cf8-bdfa-304ad5c86ad7" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703591 4821 cpu_manager.go:414] "containerMap: removing container" podUID="c0219f30-2070-4cf8-bdfa-304ad5c86ad7" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703597 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0219f30-2070-4cf8-bdfa-304ad5c86ad7" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703607 4821 cpu_manager.go:414] "containerMap: removing container" podUID="c0219f30-2070-4cf8-bdfa-304ad5c86ad7" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703613 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0219f30-2070-4cf8-bdfa-304ad5c86ad7" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703626 4821 cpu_manager.go:414] "containerMap: removing container" podUID="ff81e511-3cab-4008-b2a5-1ab9774eeff4" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703633 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff81e511-3cab-4008-b2a5-1ab9774eeff4" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703642 4821 cpu_manager.go:414] "containerMap: removing container" podUID="514b5faa-73ee-4d54-9e6b-c474a52d82ca" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703648 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="514b5faa-73ee-4d54-9e6b-c474a52d82ca" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703660 4821 cpu_manager.go:414] "containerMap: removing container" podUID="2b421f85-4894-47f3-9e87-c5b683ea4e6b" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703667 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b421f85-4894-47f3-9e87-c5b683ea4e6b" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703679 4821 cpu_manager.go:414] "containerMap: removing container" podUID="7b4b604b-3757-4c03-a745-f01e02ce82c2" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703686 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4b604b-3757-4c03-a745-f01e02ce82c2" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703703 4821 cpu_manager.go:414] "containerMap: removing container" podUID="51189101-afc0-42dd-815e-a8330ec8b42a" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703710 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51189101-afc0-42dd-815e-a8330ec8b42a" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703719 4821 cpu_manager.go:414] "containerMap: removing container" podUID="7b4b604b-3757-4c03-a745-f01e02ce82c2" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703727 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4b604b-3757-4c03-a745-f01e02ce82c2" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703738 4821 cpu_manager.go:414] "containerMap: removing container" podUID="2b421f85-4894-47f3-9e87-c5b683ea4e6b" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703745 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b421f85-4894-47f3-9e87-c5b683ea4e6b" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703950 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b421f85-4894-47f3-9e87-c5b683ea4e6b" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703963 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff81e511-3cab-4008-b2a5-1ab9774eeff4" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703971 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="137c7cf5-e922-49d1-acbc-fc0b13899da5" containerName="istio-proxy" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703981 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e44a4a-8eb9-4732-8924-d76683d1e213" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.703991 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff81e511-3cab-4008-b2a5-1ab9774eeff4" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.704002 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="786f75a2-8350-4081-8ca4-041ed3247252" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.704009 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="74bc6f99-ab50-4bd6-9b61-a8727d36f879" containerName="revoke-keys" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.704018 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="514b5faa-73ee-4d54-9e6b-c474a52d82ca" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.704027 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="51189101-afc0-42dd-815e-a8330ec8b42a" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.704035 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="786f75a2-8350-4081-8ca4-041ed3247252" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.704047 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="514b5faa-73ee-4d54-9e6b-c474a52d82ca" containerName="cleanup" Oct 29 23:15:48.704046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.704057 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="51189101-afc0-42dd-815e-a8330ec8b42a" containerName="cleanup" Oct 29 23:15:48.706402 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.704066 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0219f30-2070-4cf8-bdfa-304ad5c86ad7" containerName="cleanup" Oct 29 23:15:48.706402 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.704077 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0219f30-2070-4cf8-bdfa-304ad5c86ad7" containerName="cleanup" Oct 29 23:15:48.706402 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.704086 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0219f30-2070-4cf8-bdfa-304ad5c86ad7" containerName="cleanup" Oct 29 23:15:48.706402 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.704095 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="514b5faa-73ee-4d54-9e6b-c474a52d82ca" containerName="cleanup" Oct 29 23:15:48.706402 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.704105 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b421f85-4894-47f3-9e87-c5b683ea4e6b" containerName="cleanup" Oct 29 23:15:48.706402 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.704114 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4b604b-3757-4c03-a745-f01e02ce82c2" containerName="cleanup" Oct 29 23:15:48.706402 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.704123 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="51189101-afc0-42dd-815e-a8330ec8b42a" containerName="cleanup" Oct 29 23:15:48.706402 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.704131 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4b604b-3757-4c03-a745-f01e02ce82c2" containerName="cleanup" Oct 29 23:15:48.706402 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.704139 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b421f85-4894-47f3-9e87-c5b683ea4e6b" containerName="cleanup" Oct 29 23:15:48.706402 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.704152 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4b604b-3757-4c03-a745-f01e02ce82c2" containerName="cleanup" Oct 29 23:15:48.706402 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.704161 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a19fff5-f4be-494c-97ad-3520546346d1" containerName="main" Oct 29 23:15:48.706402 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.704170 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f71bdab-b1a2-4513-a7f1-2455e4222efa" containerName="istio-proxy" Oct 29 23:15:48.706402 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.704180 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e44a4a-8eb9-4732-8924-d76683d1e213" containerName="cleanup" Oct 29 23:15:48.706402 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.704189 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0da6fd9-efd5-4587-b764-524010b6a352" containerName="istio-proxy" Oct 29 23:15:48.710009 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.709983 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.738302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.738025 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh"] Oct 29 23:15:48.742056 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.742029 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-e170ed-openshift-default-dockercfg-lcd54\"" Oct 29 23:15:48.834232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.828111 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istiod-ca-cert\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.834232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.828564 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-podinfo\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.834232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.828622 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vm57\" (UniqueName: \"kubernetes.io/projected/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-kube-api-access-4vm57\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.834232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.828676 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-workload-certs\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.834232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.828706 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-credential-socket\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.834232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.828738 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-workload-socket\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.834232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.829046 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-envoy\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.834232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.829085 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-data\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.834232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.829289 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-token\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.930655 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.930613 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-token\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.933713 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.933684 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istiod-ca-cert\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.934727 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.934613 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istiod-ca-cert\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.935116 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.935098 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-podinfo\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.935835 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.935816 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vm57\" (UniqueName: \"kubernetes.io/projected/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-kube-api-access-4vm57\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.956302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.951053 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-workload-certs\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.956302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.951097 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-credential-socket\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.956302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.951128 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-workload-socket\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.956302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.951220 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-envoy\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.956302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.951254 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-data\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.956302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.951595 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-workload-certs\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.956302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.943583 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-podinfo\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.956302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.951817 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-data\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.956302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.951888 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-workload-socket\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.956302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.952155 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-credential-socket\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.970387 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.966802 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-envoy\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.980222 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.978058 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vm57\" (UniqueName: \"kubernetes.io/projected/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-kube-api-access-4vm57\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:48.990230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:48.981005 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-token\") pod \"e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:49.065496 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.064174 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:49.496829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.493824 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh"] Oct 29 23:15:49.499805 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.499767 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:15:49.499929 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.499852 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:15:49.499929 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.499899 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:15:49.806129 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.806033 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg"] Oct 29 23:15:49.806764 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.806740 4821 cpu_manager.go:414] "containerMap: removing container" podUID="17e44a4a-8eb9-4732-8924-d76683d1e213" containerName="cleanup" Oct 29 23:15:49.806764 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.806759 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e44a4a-8eb9-4732-8924-d76683d1e213" containerName="cleanup" Oct 29 23:15:49.806993 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.806777 4821 cpu_manager.go:414] "containerMap: removing container" podUID="51189101-afc0-42dd-815e-a8330ec8b42a" containerName="cleanup" Oct 29 23:15:49.806993 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.806783 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51189101-afc0-42dd-815e-a8330ec8b42a" containerName="cleanup" Oct 29 23:15:49.806993 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.806950 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e44a4a-8eb9-4732-8924-d76683d1e213" containerName="cleanup" Oct 29 23:15:49.806993 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.806958 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff81e511-3cab-4008-b2a5-1ab9774eeff4" containerName="cleanup" Oct 29 23:15:49.806993 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.806967 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="786f75a2-8350-4081-8ca4-041ed3247252" containerName="cleanup" Oct 29 23:15:49.810801 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.810781 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:49.828628 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.828588 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-ed3bf2-openshift-default-dockercfg-tnc4j\"" Oct 29 23:15:49.865240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.865172 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg"] Oct 29 23:15:49.921249 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.921187 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-workload-certs\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:49.921452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.921409 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-token\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:49.921532 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.921469 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkpjr\" (UniqueName: \"kubernetes.io/projected/0d738ff1-6c39-4e26-a597-fbbe75dd076b-kube-api-access-pkpjr\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:49.921601 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.921544 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-workload-socket\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:49.921661 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.921570 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istiod-ca-cert\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:49.921725 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.921673 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-data\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:49.921841 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.921816 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-credential-socket\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:49.921907 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.921865 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-envoy\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:49.921907 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:49.921886 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-podinfo\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:50.023812 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.023776 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-credential-socket\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:50.024010 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.023847 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-envoy\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:50.024010 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.023881 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-podinfo\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:50.024010 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.023968 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-workload-certs\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:50.024187 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.024120 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-token\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:50.024287 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.024188 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkpjr\" (UniqueName: \"kubernetes.io/projected/0d738ff1-6c39-4e26-a597-fbbe75dd076b-kube-api-access-pkpjr\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:50.024287 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.024255 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-workload-socket\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:50.024409 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.024284 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istiod-ca-cert\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:50.024409 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.024335 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-credential-socket\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:50.024409 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.024367 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-data\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:50.024771 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.024746 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-data\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:50.024882 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.024799 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-workload-socket\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:50.024997 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.024970 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-workload-certs\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:50.025510 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.025421 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istiod-ca-cert\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:50.027052 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.027022 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-envoy\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:50.027346 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.027328 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-podinfo\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:50.050294 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.050261 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-token\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:50.050423 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.050413 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkpjr\" (UniqueName: \"kubernetes.io/projected/0d738ff1-6c39-4e26-a597-fbbe75dd076b-kube-api-access-pkpjr\") pod \"e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:50.127702 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.127674 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:50.175820 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.175782 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" event={"ID":"36ea1460-ac2f-410a-b06a-b18f1f8e56d3","Type":"ContainerStarted","Data":"528e90721b9ce1aad59011a013291ebbf49717ddb34cf6e3b7c39895fb10beb7"} Oct 29 23:15:50.175987 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.175844 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" event={"ID":"36ea1460-ac2f-410a-b06a-b18f1f8e56d3","Type":"ContainerStarted","Data":"0879c13d04b735328dda1e6fe99c1e104cd87dff21eac5a018cfafe90ecb714c"} Oct 29 23:15:50.224570 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.224507 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" podStartSLOduration=2.224489324 podStartE2EDuration="2.224489324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:50.221899954 +0000 UTC m=+2004.162447864" watchObservedRunningTime="2026-10-29 23:15:50.224489324 +0000 UTC m=+2004.165037237" Oct 29 23:15:50.294477 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.294443 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2"] Oct 29 23:15:50.466999 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:15:50.466954 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d738ff1_6c39_4e26_a597_fbbe75dd076b.slice/crio-b217b632e9c8ce9ba3992d899fadb7f09e95a4a90e33eccc09fc0bd65d178c19 WatchSource:0}: Error finding container b217b632e9c8ce9ba3992d899fadb7f09e95a4a90e33eccc09fc0bd65d178c19: Status 404 returned error can't find the container with id b217b632e9c8ce9ba3992d899fadb7f09e95a4a90e33eccc09fc0bd65d178c19 Oct 29 23:15:50.470383 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.469147 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg"] Oct 29 23:15:50.470464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.470377 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:15:50.470464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.470439 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:15:50.470575 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:50.470471 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:15:51.067040 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:51.066956 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:51.074551 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:51.074511 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:51.186579 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:51.186540 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" event={"ID":"0d738ff1-6c39-4e26-a597-fbbe75dd076b","Type":"ContainerStarted","Data":"8c75f81d1d577b307dad77ad9f7a29e81c43b3fc4cb6e60907e55d66755da172"} Oct 29 23:15:51.186767 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:51.186601 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" event={"ID":"0d738ff1-6c39-4e26-a597-fbbe75dd076b","Type":"ContainerStarted","Data":"b217b632e9c8ce9ba3992d899fadb7f09e95a4a90e33eccc09fc0bd65d178c19"} Oct 29 23:15:51.186914 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:51.186887 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:51.188090 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:51.188059 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:15:51.245570 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:51.245505 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" podStartSLOduration=2.24548574 podStartE2EDuration="2.24548574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:51.242861642 +0000 UTC m=+2005.183409558" watchObservedRunningTime="2026-10-29 23:15:51.24548574 +0000 UTC m=+2005.186033650" Oct 29 23:15:52.132227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:52.129585 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:52.144224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:52.141855 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 29 23:15:52.144224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:52.141932 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" podUID="0d738ff1-6c39-4e26-a597-fbbe75dd076b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 29 23:15:53.128476 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:53.128442 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 29 23:15:53.128779 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:53.128750 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" podUID="0d738ff1-6c39-4e26-a597-fbbe75dd076b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 29 23:15:53.894131 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:15:53.894065 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:15:54.129227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:54.128582 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 29 23:15:54.129227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:54.128647 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" podUID="0d738ff1-6c39-4e26-a597-fbbe75dd076b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 29 23:15:54.663216 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:54.663151 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs"] Oct 29 23:15:54.668970 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:54.668940 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs" Oct 29 23:15:54.688275 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:54.688047 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs"] Oct 29 23:15:54.699429 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:54.699401 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-e170ed\"" Oct 29 23:15:54.699609 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:54.699531 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-e170ed-dockercfg-qsbpm\"" Oct 29 23:15:54.806221 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:54.800721 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b14de3-46e0-438e-9e72-68ab3c19f4b3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs\" (UID: \"65b14de3-46e0-438e-9e72-68ab3c19f4b3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs" Oct 29 23:15:54.806221 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:54.801095 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp7v9\" (UniqueName: \"kubernetes.io/projected/65b14de3-46e0-438e-9e72-68ab3c19f4b3-kube-api-access-zp7v9\") pod \"payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs\" (UID: \"65b14de3-46e0-438e-9e72-68ab3c19f4b3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs" Oct 29 23:15:54.907947 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:54.907906 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b14de3-46e0-438e-9e72-68ab3c19f4b3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs\" (UID: \"65b14de3-46e0-438e-9e72-68ab3c19f4b3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs" Oct 29 23:15:54.936915 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:54.936827 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp7v9\" (UniqueName: \"kubernetes.io/projected/65b14de3-46e0-438e-9e72-68ab3c19f4b3-kube-api-access-zp7v9\") pod \"payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs\" (UID: \"65b14de3-46e0-438e-9e72-68ab3c19f4b3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs" Oct 29 23:15:54.937714 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:54.914971 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b14de3-46e0-438e-9e72-68ab3c19f4b3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs\" (UID: \"65b14de3-46e0-438e-9e72-68ab3c19f4b3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs" Oct 29 23:15:55.108226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:55.103925 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp7v9\" (UniqueName: \"kubernetes.io/projected/65b14de3-46e0-438e-9e72-68ab3c19f4b3-kube-api-access-zp7v9\") pod \"payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs\" (UID: \"65b14de3-46e0-438e-9e72-68ab3c19f4b3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs" Oct 29 23:15:55.133387 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:55.133351 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 29 23:15:55.133662 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:55.133630 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" podUID="0d738ff1-6c39-4e26-a597-fbbe75dd076b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 29 23:15:55.294531 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:55.294440 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs" Oct 29 23:15:55.742580 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:55.742073 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs"] Oct 29 23:15:55.748465 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:15:55.748378 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b14de3_46e0_438e_9e72_68ab3c19f4b3.slice/crio-089e9f32652d1c18dd28d130e9795588da703f295a2ed711557f1c516cbd6ce5 WatchSource:0}: Error finding container 089e9f32652d1c18dd28d130e9795588da703f295a2ed711557f1c516cbd6ce5: Status 404 returned error can't find the container with id 089e9f32652d1c18dd28d130e9795588da703f295a2ed711557f1c516cbd6ce5 Oct 29 23:15:56.133227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:56.131637 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 29 23:15:56.133227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:56.131706 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" podUID="0d738ff1-6c39-4e26-a597-fbbe75dd076b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 29 23:15:56.302147 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:56.302113 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs" event={"ID":"65b14de3-46e0-438e-9e72-68ab3c19f4b3","Type":"ContainerStarted","Data":"976f9678598c769e51d94d685577369ba7c0b72c72bda7cbb013fe7424ca6ff1"} Oct 29 23:15:56.302368 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:56.302182 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs" Oct 29 23:15:56.302368 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:56.302223 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs" event={"ID":"65b14de3-46e0-438e-9e72-68ab3c19f4b3","Type":"ContainerStarted","Data":"089e9f32652d1c18dd28d130e9795588da703f295a2ed711557f1c516cbd6ce5"} Oct 29 23:15:56.361647 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:56.361584 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs" podStartSLOduration=2.361569564 podStartE2EDuration="2.361569564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:56.359301446 +0000 UTC m=+2010.299849357" watchObservedRunningTime="2026-10-29 23:15:56.361569564 +0000 UTC m=+2010.302117500" Oct 29 23:15:57.138356 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:57.138315 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 29 23:15:57.138961 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:57.138929 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" podUID="0d738ff1-6c39-4e26-a597-fbbe75dd076b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 29 23:15:58.191181 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:58.191146 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:58.191997 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:58.191959 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:15:58.193346 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:15:58.193312 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:16:05.332639 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:05.332593 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" podUID="d338e5a2-2352-492b-b76b-9b8159baa18f" containerName="main" containerID="cri-o://7cd6ff4ee8abeb0d1be0599df27b6bdc5bb82912c8750fe3e9fd2156dcaec48c" gracePeriod=45 Oct 29 23:16:05.516975 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:05.516939 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf"] Oct 29 23:16:05.861739 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:05.861707 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:16:05.970175 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:05.970140 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d338e5a2-2352-492b-b76b-9b8159baa18f-tls-certs\" (UniqueName: \"kubernetes.io/secret/d338e5a2-2352-492b-b76b-9b8159baa18f-tls-certs\") pod \"d338e5a2-2352-492b-b76b-9b8159baa18f\" (UID: \"d338e5a2-2352-492b-b76b-9b8159baa18f\") " Oct 29 23:16:05.970374 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:05.970228 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-dshm\") pod \"d338e5a2-2352-492b-b76b-9b8159baa18f\" (UID: \"d338e5a2-2352-492b-b76b-9b8159baa18f\") " Oct 29 23:16:05.970374 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:05.970282 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-model-cache\") pod \"d338e5a2-2352-492b-b76b-9b8159baa18f\" (UID: \"d338e5a2-2352-492b-b76b-9b8159baa18f\") " Oct 29 23:16:05.970509 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:05.970484 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d338e5a2-2352-492b-b76b-9b8159baa18f-kube-api-access-x5pth\" (UniqueName: \"kubernetes.io/projected/d338e5a2-2352-492b-b76b-9b8159baa18f-kube-api-access-x5pth\") pod \"d338e5a2-2352-492b-b76b-9b8159baa18f\" (UID: \"d338e5a2-2352-492b-b76b-9b8159baa18f\") " Oct 29 23:16:05.970567 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:05.970518 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-tmp-dir\") pod \"d338e5a2-2352-492b-b76b-9b8159baa18f\" (UID: \"d338e5a2-2352-492b-b76b-9b8159baa18f\") " Oct 29 23:16:05.970621 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:05.970588 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-home\" (UniqueName: \"kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-home\") pod \"d338e5a2-2352-492b-b76b-9b8159baa18f\" (UID: \"d338e5a2-2352-492b-b76b-9b8159baa18f\") " Oct 29 23:16:05.971607 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:05.971553 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-model-cache" pod "d338e5a2-2352-492b-b76b-9b8159baa18f" (UID: "d338e5a2-2352-492b-b76b-9b8159baa18f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:05.971607 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:05.971573 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-tmp-dir" pod "d338e5a2-2352-492b-b76b-9b8159baa18f" (UID: "d338e5a2-2352-492b-b76b-9b8159baa18f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:05.971786 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:05.971642 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-home" pod "d338e5a2-2352-492b-b76b-9b8159baa18f" (UID: "d338e5a2-2352-492b-b76b-9b8159baa18f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:05.973525 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:05.973490 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d338e5a2-2352-492b-b76b-9b8159baa18f-tls-certs" pod "d338e5a2-2352-492b-b76b-9b8159baa18f" (UID: "d338e5a2-2352-492b-b76b-9b8159baa18f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:16:05.973670 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:05.973622 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d338e5a2-2352-492b-b76b-9b8159baa18f-kube-api-access-x5pth" pod "d338e5a2-2352-492b-b76b-9b8159baa18f" (UID: "d338e5a2-2352-492b-b76b-9b8159baa18f"). InnerVolumeSpecName "kube-api-access-x5pth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:16:05.974280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:05.974257 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-dshm" pod "d338e5a2-2352-492b-b76b-9b8159baa18f" (UID: "d338e5a2-2352-492b-b76b-9b8159baa18f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:06.073135 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:06.073061 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d338e5a2-2352-492b-b76b-9b8159baa18f-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:06.073135 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:06.073085 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:06.073135 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:06.073093 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:06.073135 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:06.073101 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5pth\" (UniqueName: \"kubernetes.io/projected/d338e5a2-2352-492b-b76b-9b8159baa18f-kube-api-access-x5pth\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:06.073135 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:06.073111 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:06.073135 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:06.073120 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d338e5a2-2352-492b-b76b-9b8159baa18f-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:06.397356 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:06.397319 4821 generic.go:356] "Generic (PLEG): container finished" podID="d338e5a2-2352-492b-b76b-9b8159baa18f" containerID="7cd6ff4ee8abeb0d1be0599df27b6bdc5bb82912c8750fe3e9fd2156dcaec48c" exitCode=0 Oct 29 23:16:06.397814 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:06.397367 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" Oct 29 23:16:06.397814 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:06.397407 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" event={"ID":"d338e5a2-2352-492b-b76b-9b8159baa18f","Type":"ContainerDied","Data":"7cd6ff4ee8abeb0d1be0599df27b6bdc5bb82912c8750fe3e9fd2156dcaec48c"} Oct 29 23:16:06.397814 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:06.397465 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2" event={"ID":"d338e5a2-2352-492b-b76b-9b8159baa18f","Type":"ContainerDied","Data":"ff9aaabe3eaffbe5b698c3771ed7ddc0c4858e6622172e7c272cc8c71613e295"} Oct 29 23:16:06.397814 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:06.397491 4821 scope.go:122] "RemoveContainer" containerID="7cd6ff4ee8abeb0d1be0599df27b6bdc5bb82912c8750fe3e9fd2156dcaec48c" Oct 29 23:16:06.409440 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:06.409420 4821 scope.go:122] "RemoveContainer" containerID="7cd6ff4ee8abeb0d1be0599df27b6bdc5bb82912c8750fe3e9fd2156dcaec48c" Oct 29 23:16:06.409747 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:16:06.409726 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd6ff4ee8abeb0d1be0599df27b6bdc5bb82912c8750fe3e9fd2156dcaec48c\": container with ID starting with 7cd6ff4ee8abeb0d1be0599df27b6bdc5bb82912c8750fe3e9fd2156dcaec48c not found: ID does not exist" containerID="7cd6ff4ee8abeb0d1be0599df27b6bdc5bb82912c8750fe3e9fd2156dcaec48c" Oct 29 23:16:06.409795 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:06.409755 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd6ff4ee8abeb0d1be0599df27b6bdc5bb82912c8750fe3e9fd2156dcaec48c"} err="failed to get container status \"7cd6ff4ee8abeb0d1be0599df27b6bdc5bb82912c8750fe3e9fd2156dcaec48c\": rpc error: code = NotFound desc = could not find container \"7cd6ff4ee8abeb0d1be0599df27b6bdc5bb82912c8750fe3e9fd2156dcaec48c\": container with ID starting with 7cd6ff4ee8abeb0d1be0599df27b6bdc5bb82912c8750fe3e9fd2156dcaec48c not found: ID does not exist" Oct 29 23:16:06.456581 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:06.456548 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2"] Oct 29 23:16:06.468573 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:06.468537 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-65e2d3/auth-test-model-65e2d3-kserve-5fc9f49785-2mfg2"] Oct 29 23:16:07.313848 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:07.313821 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs" Oct 29 23:16:08.173866 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:08.173833 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d338e5a2-2352-492b-b76b-9b8159baa18f" path="/var/lib/kubelet/pods/d338e5a2-2352-492b-b76b-9b8159baa18f/volumes" Oct 29 23:16:12.268009 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:12.267974 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq"] Oct 29 23:16:12.268727 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:12.268710 4821 cpu_manager.go:414] "containerMap: removing container" podUID="d338e5a2-2352-492b-b76b-9b8159baa18f" containerName="main" Oct 29 23:16:12.268778 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:12.268729 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d338e5a2-2352-492b-b76b-9b8159baa18f" containerName="main" Oct 29 23:16:12.268843 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:12.268834 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="d338e5a2-2352-492b-b76b-9b8159baa18f" containerName="main" Oct 29 23:16:12.273392 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:12.273374 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq" Oct 29 23:16:12.291115 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:12.291085 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq"] Oct 29 23:16:12.298812 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:12.298787 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-faa4fc\"" Oct 29 23:16:12.304559 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:12.298826 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-faa4fc-dockercfg-5k98q\"" Oct 29 23:16:12.308542 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:12.308508 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c4bd01-02f4-4c07-8792-4f880626a8c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq\" (UID: \"27c4bd01-02f4-4c07-8792-4f880626a8c0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq" Oct 29 23:16:12.308704 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:12.308637 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xncdq\" (UniqueName: \"kubernetes.io/projected/27c4bd01-02f4-4c07-8792-4f880626a8c0-kube-api-access-xncdq\") pod \"payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq\" (UID: \"27c4bd01-02f4-4c07-8792-4f880626a8c0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq" Oct 29 23:16:12.415484 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:12.415282 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xncdq\" (UniqueName: \"kubernetes.io/projected/27c4bd01-02f4-4c07-8792-4f880626a8c0-kube-api-access-xncdq\") pod \"payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq\" (UID: \"27c4bd01-02f4-4c07-8792-4f880626a8c0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq" Oct 29 23:16:12.416635 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:12.416606 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c4bd01-02f4-4c07-8792-4f880626a8c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq\" (UID: \"27c4bd01-02f4-4c07-8792-4f880626a8c0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq" Oct 29 23:16:12.417603 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:12.417569 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c4bd01-02f4-4c07-8792-4f880626a8c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq\" (UID: \"27c4bd01-02f4-4c07-8792-4f880626a8c0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq" Oct 29 23:16:12.648226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:12.641756 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xncdq\" (UniqueName: \"kubernetes.io/projected/27c4bd01-02f4-4c07-8792-4f880626a8c0-kube-api-access-xncdq\") pod \"payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq\" (UID: \"27c4bd01-02f4-4c07-8792-4f880626a8c0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq" Oct 29 23:16:12.893012 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:12.892979 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq" Oct 29 23:16:13.209341 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:13.209311 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq"] Oct 29 23:16:13.210006 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:16:13.209981 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c4bd01_02f4_4c07_8792_4f880626a8c0.slice/crio-a5291477216430493a0a4556b4016e4b0d6bfaf42439de8c2d981247e2664568 WatchSource:0}: Error finding container a5291477216430493a0a4556b4016e4b0d6bfaf42439de8c2d981247e2664568: Status 404 returned error can't find the container with id a5291477216430493a0a4556b4016e4b0d6bfaf42439de8c2d981247e2664568 Oct 29 23:16:13.587282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:13.587141 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq" event={"ID":"27c4bd01-02f4-4c07-8792-4f880626a8c0","Type":"ContainerStarted","Data":"d741cd0c92a2e9aefdc43d89d9a4b4908b37a9ecff36883fb2eab788a02d9595"} Oct 29 23:16:13.587282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:13.587262 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq" event={"ID":"27c4bd01-02f4-4c07-8792-4f880626a8c0","Type":"ContainerStarted","Data":"a5291477216430493a0a4556b4016e4b0d6bfaf42439de8c2d981247e2664568"} Oct 29 23:16:13.587851 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:13.587537 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq" Oct 29 23:16:13.635329 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:13.635263 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq" podStartSLOduration=1.635242358 podStartE2EDuration="1.635242358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:13.62972909 +0000 UTC m=+2027.570277013" watchObservedRunningTime="2026-10-29 23:16:13.635242358 +0000 UTC m=+2027.575790269" Oct 29 23:16:20.548924 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:20.548878 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" podUID="2aeb91d0-8e92-4c73-ae62-767cdb2e6f84" containerName="main" containerID="cri-o://8befdf747f801f2cad604269aaf9fa4a11c4509dbe4a4e3ff4f9d40c66616b8a" gracePeriod=45 Oct 29 23:16:20.911136 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:20.911109 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:16:21.051581 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.051552 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-dshm\") pod \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\" (UID: \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\") " Oct 29 23:16:21.051581 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.051585 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-model-cache\") pod \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\" (UID: \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\") " Oct 29 23:16:21.051830 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.051619 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-tmp-dir\") pod \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\" (UID: \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\") " Oct 29 23:16:21.051830 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.051650 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-kube-api-access-ttszj\" (UniqueName: \"kubernetes.io/projected/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-kube-api-access-ttszj\") pod \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\" (UID: \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\") " Oct 29 23:16:21.051830 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.051725 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-home\" (UniqueName: \"kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-home\") pod \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\" (UID: \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\") " Oct 29 23:16:21.052013 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.051911 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-model-cache" pod "2aeb91d0-8e92-4c73-ae62-767cdb2e6f84" (UID: "2aeb91d0-8e92-4c73-ae62-767cdb2e6f84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:21.052013 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.051945 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-tls-certs\" (UniqueName: \"kubernetes.io/secret/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-tls-certs\") pod \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\" (UID: \"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84\") " Oct 29 23:16:21.052158 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.052117 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-tmp-dir" pod "2aeb91d0-8e92-4c73-ae62-767cdb2e6f84" (UID: "2aeb91d0-8e92-4c73-ae62-767cdb2e6f84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:21.052575 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.052489 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-home" pod "2aeb91d0-8e92-4c73-ae62-767cdb2e6f84" (UID: "2aeb91d0-8e92-4c73-ae62-767cdb2e6f84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:21.053060 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.053036 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:21.053174 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.053090 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:21.053174 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.053105 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:21.054376 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.054353 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-dshm" pod "2aeb91d0-8e92-4c73-ae62-767cdb2e6f84" (UID: "2aeb91d0-8e92-4c73-ae62-767cdb2e6f84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:21.054499 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.054447 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-tls-certs" pod "2aeb91d0-8e92-4c73-ae62-767cdb2e6f84" (UID: "2aeb91d0-8e92-4c73-ae62-767cdb2e6f84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:16:21.054499 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.054467 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-kube-api-access-ttszj" pod "2aeb91d0-8e92-4c73-ae62-767cdb2e6f84" (UID: "2aeb91d0-8e92-4c73-ae62-767cdb2e6f84"). InnerVolumeSpecName "kube-api-access-ttszj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:16:21.155012 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.154940 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttszj\" (UniqueName: \"kubernetes.io/projected/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-kube-api-access-ttszj\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:21.155012 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.154966 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:21.155012 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.154976 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:21.657386 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.657352 4821 generic.go:356] "Generic (PLEG): container finished" podID="2aeb91d0-8e92-4c73-ae62-767cdb2e6f84" containerID="8befdf747f801f2cad604269aaf9fa4a11c4509dbe4a4e3ff4f9d40c66616b8a" exitCode=0 Oct 29 23:16:21.658123 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.657425 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" Oct 29 23:16:21.658123 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.657445 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" event={"ID":"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84","Type":"ContainerDied","Data":"8befdf747f801f2cad604269aaf9fa4a11c4509dbe4a4e3ff4f9d40c66616b8a"} Oct 29 23:16:21.658123 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.657595 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf" event={"ID":"2aeb91d0-8e92-4c73-ae62-767cdb2e6f84","Type":"ContainerDied","Data":"38a0b357d1680e2e25e367552289d827c61470446ed4727454ea9593fb1480b3"} Oct 29 23:16:21.658123 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.657619 4821 scope.go:122] "RemoveContainer" containerID="8befdf747f801f2cad604269aaf9fa4a11c4509dbe4a4e3ff4f9d40c66616b8a" Oct 29 23:16:21.669679 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.669661 4821 scope.go:122] "RemoveContainer" containerID="8befdf747f801f2cad604269aaf9fa4a11c4509dbe4a4e3ff4f9d40c66616b8a" Oct 29 23:16:21.669963 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:16:21.669942 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8befdf747f801f2cad604269aaf9fa4a11c4509dbe4a4e3ff4f9d40c66616b8a\": container with ID starting with 8befdf747f801f2cad604269aaf9fa4a11c4509dbe4a4e3ff4f9d40c66616b8a not found: ID does not exist" containerID="8befdf747f801f2cad604269aaf9fa4a11c4509dbe4a4e3ff4f9d40c66616b8a" Oct 29 23:16:21.670048 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.669971 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8befdf747f801f2cad604269aaf9fa4a11c4509dbe4a4e3ff4f9d40c66616b8a"} err="failed to get container status \"8befdf747f801f2cad604269aaf9fa4a11c4509dbe4a4e3ff4f9d40c66616b8a\": rpc error: code = NotFound desc = could not find container \"8befdf747f801f2cad604269aaf9fa4a11c4509dbe4a4e3ff4f9d40c66616b8a\": container with ID starting with 8befdf747f801f2cad604269aaf9fa4a11c4509dbe4a4e3ff4f9d40c66616b8a not found: ID does not exist" Oct 29 23:16:21.715535 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.715500 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf"] Oct 29 23:16:21.728471 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:21.728439 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-755453/auth-test-model-755453-kserve-75c564dd45-98jzf"] Oct 29 23:16:22.172433 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:22.172388 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aeb91d0-8e92-4c73-ae62-767cdb2e6f84" path="/var/lib/kubelet/pods/2aeb91d0-8e92-4c73-ae62-767cdb2e6f84/volumes" Oct 29 23:16:23.129225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:23.115567 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7"] Oct 29 23:16:23.129225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:23.118993 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84"] Oct 29 23:16:23.129225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:23.119269 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" podUID="e6f4fd46-5aab-49a8-9446-f71ed103d2eb" containerName="istio-proxy" containerID="cri-o://f49f74bb323fdca15e5b27b1a0298fbbbea9b4e670b9c83fd9c6062441e33602" gracePeriod=30 Oct 29 23:16:23.129225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:23.126767 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn"] Oct 29 23:16:23.129225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:23.126978 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" podUID="62c0101f-490c-470e-b945-e3bb7c86d2c6" containerName="istio-proxy" containerID="cri-o://5a94f480d920d388d547679f958ac5acc83b198e28d5b7c50c08f7c7211b1975" gracePeriod=30 Oct 29 23:16:23.144986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:23.142063 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght"] Oct 29 23:16:23.144986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:23.142142 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s"] Oct 29 23:16:23.144986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:23.142342 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" podUID="041cb60f-5e7c-47d7-8e83-cce35d4c9529" containerName="istio-proxy" containerID="cri-o://927a14a66db06c1f045de17016d17f867b327016462e4f2f84519c9643b62ac4" gracePeriod=30 Oct 29 23:16:23.144986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:23.142577 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" podUID="e02c68ae-98e8-47ce-8b21-9aa5025a20d5" containerName="istio-proxy" containerID="cri-o://e14e2622fb7b7c26fab34b5caa08a7d0d83b38b2e2728f1fbf0af8d90e7c0bd7" gracePeriod=30 Oct 29 23:16:23.891263 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:16:23.891223 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:16:24.614801 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:24.614772 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq" Oct 29 23:16:25.919973 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:25.916003 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 23:16:25.964225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:25.959281 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:16:28.020453 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.020415 4821 patch_prober.go:29] interesting pod/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:16:28.021107 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.021073 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" podUID="e6f4fd46-5aab-49a8-9446-f71ed103d2eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:16:28.245827 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.245785 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp"] Oct 29 23:16:28.681139 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.681114 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:16:28.706671 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.706645 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 23:16:28.809272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.807016 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-kube-api-access-5q5wl\" (UniqueName: \"kubernetes.io/projected/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-kube-api-access-5q5wl\") pod \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " Oct 29 23:16:28.809272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.807079 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-workload-certs\") pod \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " Oct 29 23:16:28.809272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.807148 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-workload-certs\") pod \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " Oct 29 23:16:28.809272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.807179 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-data\") pod \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " Oct 29 23:16:28.809272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.807235 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-workload-socket\") pod \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " Oct 29 23:16:28.809272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.807273 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-podinfo\") pod \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " Oct 29 23:16:28.809272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.807320 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-credential-socket\") pod \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " Oct 29 23:16:28.809272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.807415 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-data\") pod \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " Oct 29 23:16:28.809272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.807448 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-podinfo\") pod \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " Oct 29 23:16:28.809272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.807510 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istiod-ca-cert\") pod \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " Oct 29 23:16:28.809272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.807516 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-workload-certs" pod "e02c68ae-98e8-47ce-8b21-9aa5025a20d5" (UID: "e02c68ae-98e8-47ce-8b21-9aa5025a20d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:28.809272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.807538 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-envoy\") pod \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " Oct 29 23:16:28.809272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.807584 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-token\" (UniqueName: \"kubernetes.io/projected/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-token\") pod \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " Oct 29 23:16:28.809272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.807615 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-credential-socket\") pod \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " Oct 29 23:16:28.809272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.807656 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istiod-ca-cert\") pod \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " Oct 29 23:16:28.809272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.807741 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-envoy\") pod \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " Oct 29 23:16:28.809272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.807780 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/041cb60f-5e7c-47d7-8e83-cce35d4c9529-kube-api-access-2zmf2\" (UniqueName: \"kubernetes.io/projected/041cb60f-5e7c-47d7-8e83-cce35d4c9529-kube-api-access-2zmf2\") pod \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\" (UID: \"041cb60f-5e7c-47d7-8e83-cce35d4c9529\") " Oct 29 23:16:28.809272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.807854 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-workload-socket\") pod \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " Oct 29 23:16:28.809272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.807922 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-token\" (UniqueName: \"kubernetes.io/projected/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-token\") pod \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\" (UID: \"e02c68ae-98e8-47ce-8b21-9aa5025a20d5\") " Oct 29 23:16:28.809272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.808580 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istiod-ca-cert" pod "e02c68ae-98e8-47ce-8b21-9aa5025a20d5" (UID: "e02c68ae-98e8-47ce-8b21-9aa5025a20d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:16:28.809272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.808803 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-data" pod "e02c68ae-98e8-47ce-8b21-9aa5025a20d5" (UID: "e02c68ae-98e8-47ce-8b21-9aa5025a20d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:28.809272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.809031 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-workload-socket" pod "041cb60f-5e7c-47d7-8e83-cce35d4c9529" (UID: "041cb60f-5e7c-47d7-8e83-cce35d4c9529"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:28.809272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.809049 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-credential-socket" pod "e02c68ae-98e8-47ce-8b21-9aa5025a20d5" (UID: "e02c68ae-98e8-47ce-8b21-9aa5025a20d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:28.810695 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.809386 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-data" pod "041cb60f-5e7c-47d7-8e83-cce35d4c9529" (UID: "041cb60f-5e7c-47d7-8e83-cce35d4c9529"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:28.810695 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.809502 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-workload-socket" pod "e02c68ae-98e8-47ce-8b21-9aa5025a20d5" (UID: "e02c68ae-98e8-47ce-8b21-9aa5025a20d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:28.810695 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.809549 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-workload-certs" pod "041cb60f-5e7c-47d7-8e83-cce35d4c9529" (UID: "041cb60f-5e7c-47d7-8e83-cce35d4c9529"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:28.810695 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.809750 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-credential-socket" pod "041cb60f-5e7c-47d7-8e83-cce35d4c9529" (UID: "041cb60f-5e7c-47d7-8e83-cce35d4c9529"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:28.810695 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.810547 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:28.810695 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.810572 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:28.810695 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.810590 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:28.810695 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.810603 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:28.810695 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.810617 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:28.810695 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.810634 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:28.810695 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.810648 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:28.810695 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.810662 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:28.810695 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.810675 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:28.811567 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.811103 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istiod-ca-cert" pod "041cb60f-5e7c-47d7-8e83-cce35d4c9529" (UID: "041cb60f-5e7c-47d7-8e83-cce35d4c9529"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:16:28.812524 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.812495 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-envoy" pod "e02c68ae-98e8-47ce-8b21-9aa5025a20d5" (UID: "e02c68ae-98e8-47ce-8b21-9aa5025a20d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:28.814466 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.814441 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-podinfo" pod "e02c68ae-98e8-47ce-8b21-9aa5025a20d5" (UID: "e02c68ae-98e8-47ce-8b21-9aa5025a20d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:16:28.814614 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.814454 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-envoy" pod "041cb60f-5e7c-47d7-8e83-cce35d4c9529" (UID: "041cb60f-5e7c-47d7-8e83-cce35d4c9529"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:28.814838 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.814809 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-token" pod "041cb60f-5e7c-47d7-8e83-cce35d4c9529" (UID: "041cb60f-5e7c-47d7-8e83-cce35d4c9529"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:16:28.814924 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.814870 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-token" pod "e02c68ae-98e8-47ce-8b21-9aa5025a20d5" (UID: "e02c68ae-98e8-47ce-8b21-9aa5025a20d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:16:28.815864 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.815828 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-kube-api-access-5q5wl" pod "e02c68ae-98e8-47ce-8b21-9aa5025a20d5" (UID: "e02c68ae-98e8-47ce-8b21-9aa5025a20d5"). InnerVolumeSpecName "kube-api-access-5q5wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:16:28.820266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.820168 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/041cb60f-5e7c-47d7-8e83-cce35d4c9529-kube-api-access-2zmf2" pod "041cb60f-5e7c-47d7-8e83-cce35d4c9529" (UID: "041cb60f-5e7c-47d7-8e83-cce35d4c9529"). InnerVolumeSpecName "kube-api-access-2zmf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:16:28.820683 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.820659 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-podinfo" pod "041cb60f-5e7c-47d7-8e83-cce35d4c9529" (UID: "041cb60f-5e7c-47d7-8e83-cce35d4c9529"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:16:28.913261 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.913228 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:28.913543 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.913528 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:28.913622 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.913613 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:28.913702 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.913694 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:28.913763 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.913753 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zmf2\" (UniqueName: \"kubernetes.io/projected/041cb60f-5e7c-47d7-8e83-cce35d4c9529-kube-api-access-2zmf2\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:28.913828 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.913820 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:28.915313 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.915293 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q5wl\" (UniqueName: \"kubernetes.io/projected/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-kube-api-access-5q5wl\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:28.915507 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.915489 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e02c68ae-98e8-47ce-8b21-9aa5025a20d5-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:28.915671 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.915656 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/041cb60f-5e7c-47d7-8e83-cce35d4c9529-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:28.958375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.958346 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 23:16:28.975574 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:28.975548 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 23:16:29.098399 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.098361 4821 generic.go:356] "Generic (PLEG): container finished" podID="e02c68ae-98e8-47ce-8b21-9aa5025a20d5" containerID="e14e2622fb7b7c26fab34b5caa08a7d0d83b38b2e2728f1fbf0af8d90e7c0bd7" exitCode=0 Oct 29 23:16:29.098879 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.098423 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" Oct 29 23:16:29.098879 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.098464 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" event={"ID":"e02c68ae-98e8-47ce-8b21-9aa5025a20d5","Type":"ContainerDied","Data":"e14e2622fb7b7c26fab34b5caa08a7d0d83b38b2e2728f1fbf0af8d90e7c0bd7"} Oct 29 23:16:29.099096 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.099069 4821 scope.go:122] "RemoveContainer" containerID="e14e2622fb7b7c26fab34b5caa08a7d0d83b38b2e2728f1fbf0af8d90e7c0bd7" Oct 29 23:16:29.099286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.098986 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght" event={"ID":"e02c68ae-98e8-47ce-8b21-9aa5025a20d5","Type":"ContainerDied","Data":"bcae21405961c82c7b9efc0be67ba3054fcae2f30838ad2a071f965793a7144b"} Oct 29 23:16:29.100736 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.100710 4821 generic.go:356] "Generic (PLEG): container finished" podID="62c0101f-490c-470e-b945-e3bb7c86d2c6" containerID="5a94f480d920d388d547679f958ac5acc83b198e28d5b7c50c08f7c7211b1975" exitCode=0 Oct 29 23:16:29.100851 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.100819 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" event={"ID":"62c0101f-490c-470e-b945-e3bb7c86d2c6","Type":"ContainerDied","Data":"5a94f480d920d388d547679f958ac5acc83b198e28d5b7c50c08f7c7211b1975"} Oct 29 23:16:29.101002 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.100854 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" event={"ID":"62c0101f-490c-470e-b945-e3bb7c86d2c6","Type":"ContainerDied","Data":"de32924776c41532e8c864cf101ec71c33699162edce8043b3fb6a11e0edd862"} Oct 29 23:16:29.101121 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.101103 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn" Oct 29 23:16:29.103672 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.103647 4821 generic.go:356] "Generic (PLEG): container finished" podID="e6f4fd46-5aab-49a8-9446-f71ed103d2eb" containerID="f49f74bb323fdca15e5b27b1a0298fbbbea9b4e670b9c83fd9c6062441e33602" exitCode=0 Oct 29 23:16:29.103798 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.103729 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" Oct 29 23:16:29.103874 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.103804 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" event={"ID":"e6f4fd46-5aab-49a8-9446-f71ed103d2eb","Type":"ContainerDied","Data":"f49f74bb323fdca15e5b27b1a0298fbbbea9b4e670b9c83fd9c6062441e33602"} Oct 29 23:16:29.103874 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.103843 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84" event={"ID":"e6f4fd46-5aab-49a8-9446-f71ed103d2eb","Type":"ContainerDied","Data":"bea33b5f18c7ed547384ac035f02e14fe69bea76c1fe5f35236dc0631db46509"} Oct 29 23:16:29.106233 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.106171 4821 generic.go:356] "Generic (PLEG): container finished" podID="041cb60f-5e7c-47d7-8e83-cce35d4c9529" containerID="927a14a66db06c1f045de17016d17f867b327016462e4f2f84519c9643b62ac4" exitCode=0 Oct 29 23:16:29.106446 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.106409 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" event={"ID":"041cb60f-5e7c-47d7-8e83-cce35d4c9529","Type":"ContainerDied","Data":"927a14a66db06c1f045de17016d17f867b327016462e4f2f84519c9643b62ac4"} Oct 29 23:16:29.106736 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.106707 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" event={"ID":"041cb60f-5e7c-47d7-8e83-cce35d4c9529","Type":"ContainerDied","Data":"49e9848d0d000a6d4d2d0b9546a180fbcd19caebbc2d490c9bbbfec26624e0e8"} Oct 29 23:16:29.106923 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.106521 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s" Oct 29 23:16:29.118619 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.118593 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-workload-certs\") pod \"62c0101f-490c-470e-b945-e3bb7c86d2c6\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " Oct 29 23:16:29.118737 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.118640 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-workload-certs\") pod \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " Oct 29 23:16:29.118737 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.118683 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-token\" (UniqueName: \"kubernetes.io/projected/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-token\") pod \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " Oct 29 23:16:29.118737 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.118710 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istiod-ca-cert\") pod \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " Oct 29 23:16:29.118901 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.118786 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-credential-socket\") pod \"62c0101f-490c-470e-b945-e3bb7c86d2c6\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " Oct 29 23:16:29.118901 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.118818 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-token\" (UniqueName: \"kubernetes.io/projected/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-token\") pod \"62c0101f-490c-470e-b945-e3bb7c86d2c6\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " Oct 29 23:16:29.118901 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.118856 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-data\") pod \"62c0101f-490c-470e-b945-e3bb7c86d2c6\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " Oct 29 23:16:29.118901 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.118844 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-workload-certs" pod "62c0101f-490c-470e-b945-e3bb7c86d2c6" (UID: "62c0101f-490c-470e-b945-e3bb7c86d2c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:29.119118 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.118919 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-workload-certs" pod "e6f4fd46-5aab-49a8-9446-f71ed103d2eb" (UID: "e6f4fd46-5aab-49a8-9446-f71ed103d2eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:29.119118 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.118933 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-workload-socket\") pod \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " Oct 29 23:16:29.119118 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.118965 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-kube-api-access-6m2h7\" (UniqueName: \"kubernetes.io/projected/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-kube-api-access-6m2h7\") pod \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " Oct 29 23:16:29.119118 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.119002 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-workload-socket\") pod \"62c0101f-490c-470e-b945-e3bb7c86d2c6\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " Oct 29 23:16:29.119118 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.119048 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-data\") pod \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " Oct 29 23:16:29.119118 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.119066 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-credential-socket" pod "62c0101f-490c-470e-b945-e3bb7c86d2c6" (UID: "62c0101f-490c-470e-b945-e3bb7c86d2c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:29.119482 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.119143 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c0101f-490c-470e-b945-e3bb7c86d2c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c0101f-490c-470e-b945-e3bb7c86d2c6-istiod-ca-cert\") pod \"62c0101f-490c-470e-b945-e3bb7c86d2c6\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " Oct 29 23:16:29.119482 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.119159 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-data" pod "62c0101f-490c-470e-b945-e3bb7c86d2c6" (UID: "62c0101f-490c-470e-b945-e3bb7c86d2c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:29.119482 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.119178 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-podinfo\") pod \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " Oct 29 23:16:29.119482 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.119295 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-workload-socket" pod "e6f4fd46-5aab-49a8-9446-f71ed103d2eb" (UID: "e6f4fd46-5aab-49a8-9446-f71ed103d2eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:29.119482 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.119321 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-workload-socket" pod "62c0101f-490c-470e-b945-e3bb7c86d2c6" (UID: "62c0101f-490c-470e-b945-e3bb7c86d2c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:29.119482 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.119323 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-podinfo\") pod \"62c0101f-490c-470e-b945-e3bb7c86d2c6\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " Oct 29 23:16:29.119482 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.119432 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-envoy\") pod \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " Oct 29 23:16:29.119949 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.119492 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-credential-socket\") pod \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\" (UID: \"e6f4fd46-5aab-49a8-9446-f71ed103d2eb\") " Oct 29 23:16:29.119949 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.119572 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-envoy\") pod \"62c0101f-490c-470e-b945-e3bb7c86d2c6\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " Oct 29 23:16:29.119949 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.119625 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c0101f-490c-470e-b945-e3bb7c86d2c6-istiod-ca-cert" pod "62c0101f-490c-470e-b945-e3bb7c86d2c6" (UID: "62c0101f-490c-470e-b945-e3bb7c86d2c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:16:29.119949 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.119646 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c0101f-490c-470e-b945-e3bb7c86d2c6-kube-api-access-ctv5z\" (UniqueName: \"kubernetes.io/projected/62c0101f-490c-470e-b945-e3bb7c86d2c6-kube-api-access-ctv5z\") pod \"62c0101f-490c-470e-b945-e3bb7c86d2c6\" (UID: \"62c0101f-490c-470e-b945-e3bb7c86d2c6\") " Oct 29 23:16:29.119949 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.119842 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istiod-ca-cert" pod "e6f4fd46-5aab-49a8-9446-f71ed103d2eb" (UID: "e6f4fd46-5aab-49a8-9446-f71ed103d2eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:16:29.119949 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.119850 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-credential-socket" pod "e6f4fd46-5aab-49a8-9446-f71ed103d2eb" (UID: "e6f4fd46-5aab-49a8-9446-f71ed103d2eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:29.119949 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.119923 4821 scope.go:122] "RemoveContainer" containerID="e14e2622fb7b7c26fab34b5caa08a7d0d83b38b2e2728f1fbf0af8d90e7c0bd7" Oct 29 23:16:29.120507 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:16:29.120485 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e14e2622fb7b7c26fab34b5caa08a7d0d83b38b2e2728f1fbf0af8d90e7c0bd7\": container with ID starting with e14e2622fb7b7c26fab34b5caa08a7d0d83b38b2e2728f1fbf0af8d90e7c0bd7 not found: ID does not exist" containerID="e14e2622fb7b7c26fab34b5caa08a7d0d83b38b2e2728f1fbf0af8d90e7c0bd7" Oct 29 23:16:29.120605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.120519 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e14e2622fb7b7c26fab34b5caa08a7d0d83b38b2e2728f1fbf0af8d90e7c0bd7"} err="failed to get container status \"e14e2622fb7b7c26fab34b5caa08a7d0d83b38b2e2728f1fbf0af8d90e7c0bd7\": rpc error: code = NotFound desc = could not find container \"e14e2622fb7b7c26fab34b5caa08a7d0d83b38b2e2728f1fbf0af8d90e7c0bd7\": container with ID starting with e14e2622fb7b7c26fab34b5caa08a7d0d83b38b2e2728f1fbf0af8d90e7c0bd7 not found: ID does not exist" Oct 29 23:16:29.120605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.120544 4821 scope.go:122] "RemoveContainer" containerID="5a94f480d920d388d547679f958ac5acc83b198e28d5b7c50c08f7c7211b1975" Oct 29 23:16:29.120605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.120554 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:29.120827 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.120582 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:29.120827 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.120644 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:29.120827 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.120657 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:29.120827 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.120671 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:29.120827 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.120684 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:29.120827 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.120698 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:29.120827 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.120711 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c0101f-490c-470e-b945-e3bb7c86d2c6-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:29.120827 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.120622 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-data" pod "e6f4fd46-5aab-49a8-9446-f71ed103d2eb" (UID: "e6f4fd46-5aab-49a8-9446-f71ed103d2eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:29.120827 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.120725 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:29.129607 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.129549 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-token" pod "e6f4fd46-5aab-49a8-9446-f71ed103d2eb" (UID: "e6f4fd46-5aab-49a8-9446-f71ed103d2eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:16:29.133363 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.133339 4821 scope.go:122] "RemoveContainer" containerID="5a94f480d920d388d547679f958ac5acc83b198e28d5b7c50c08f7c7211b1975" Oct 29 23:16:29.133677 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:16:29.133656 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a94f480d920d388d547679f958ac5acc83b198e28d5b7c50c08f7c7211b1975\": container with ID starting with 5a94f480d920d388d547679f958ac5acc83b198e28d5b7c50c08f7c7211b1975 not found: ID does not exist" containerID="5a94f480d920d388d547679f958ac5acc83b198e28d5b7c50c08f7c7211b1975" Oct 29 23:16:29.133758 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.133684 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a94f480d920d388d547679f958ac5acc83b198e28d5b7c50c08f7c7211b1975"} err="failed to get container status \"5a94f480d920d388d547679f958ac5acc83b198e28d5b7c50c08f7c7211b1975\": rpc error: code = NotFound desc = could not find container \"5a94f480d920d388d547679f958ac5acc83b198e28d5b7c50c08f7c7211b1975\": container with ID starting with 5a94f480d920d388d547679f958ac5acc83b198e28d5b7c50c08f7c7211b1975 not found: ID does not exist" Oct 29 23:16:29.133758 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.133699 4821 scope.go:122] "RemoveContainer" containerID="f49f74bb323fdca15e5b27b1a0298fbbbea9b4e670b9c83fd9c6062441e33602" Oct 29 23:16:29.139658 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.139636 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-kube-api-access-6m2h7" pod "e6f4fd46-5aab-49a8-9446-f71ed103d2eb" (UID: "e6f4fd46-5aab-49a8-9446-f71ed103d2eb"). InnerVolumeSpecName "kube-api-access-6m2h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:16:29.139658 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.139645 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-envoy" pod "e6f4fd46-5aab-49a8-9446-f71ed103d2eb" (UID: "e6f4fd46-5aab-49a8-9446-f71ed103d2eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:29.139820 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.139679 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-podinfo" pod "e6f4fd46-5aab-49a8-9446-f71ed103d2eb" (UID: "e6f4fd46-5aab-49a8-9446-f71ed103d2eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:16:29.139820 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.139719 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-token" pod "62c0101f-490c-470e-b945-e3bb7c86d2c6" (UID: "62c0101f-490c-470e-b945-e3bb7c86d2c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:16:29.139820 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.139758 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-envoy" pod "62c0101f-490c-470e-b945-e3bb7c86d2c6" (UID: "62c0101f-490c-470e-b945-e3bb7c86d2c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:29.139820 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.139754 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c0101f-490c-470e-b945-e3bb7c86d2c6-kube-api-access-ctv5z" pod "62c0101f-490c-470e-b945-e3bb7c86d2c6" (UID: "62c0101f-490c-470e-b945-e3bb7c86d2c6"). InnerVolumeSpecName "kube-api-access-ctv5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:16:29.139820 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.139793 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-podinfo" pod "62c0101f-490c-470e-b945-e3bb7c86d2c6" (UID: "62c0101f-490c-470e-b945-e3bb7c86d2c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:16:29.144867 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.144849 4821 scope.go:122] "RemoveContainer" containerID="f49f74bb323fdca15e5b27b1a0298fbbbea9b4e670b9c83fd9c6062441e33602" Oct 29 23:16:29.145152 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:16:29.145134 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f49f74bb323fdca15e5b27b1a0298fbbbea9b4e670b9c83fd9c6062441e33602\": container with ID starting with f49f74bb323fdca15e5b27b1a0298fbbbea9b4e670b9c83fd9c6062441e33602 not found: ID does not exist" containerID="f49f74bb323fdca15e5b27b1a0298fbbbea9b4e670b9c83fd9c6062441e33602" Oct 29 23:16:29.145286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.145161 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f49f74bb323fdca15e5b27b1a0298fbbbea9b4e670b9c83fd9c6062441e33602"} err="failed to get container status \"f49f74bb323fdca15e5b27b1a0298fbbbea9b4e670b9c83fd9c6062441e33602\": rpc error: code = NotFound desc = could not find container \"f49f74bb323fdca15e5b27b1a0298fbbbea9b4e670b9c83fd9c6062441e33602\": container with ID starting with f49f74bb323fdca15e5b27b1a0298fbbbea9b4e670b9c83fd9c6062441e33602 not found: ID does not exist" Oct 29 23:16:29.145286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.145179 4821 scope.go:122] "RemoveContainer" containerID="927a14a66db06c1f045de17016d17f867b327016462e4f2f84519c9643b62ac4" Oct 29 23:16:29.154790 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.154772 4821 scope.go:122] "RemoveContainer" containerID="927a14a66db06c1f045de17016d17f867b327016462e4f2f84519c9643b62ac4" Oct 29 23:16:29.155062 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:16:29.155045 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"927a14a66db06c1f045de17016d17f867b327016462e4f2f84519c9643b62ac4\": container with ID starting with 927a14a66db06c1f045de17016d17f867b327016462e4f2f84519c9643b62ac4 not found: ID does not exist" containerID="927a14a66db06c1f045de17016d17f867b327016462e4f2f84519c9643b62ac4" Oct 29 23:16:29.155114 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.155068 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"927a14a66db06c1f045de17016d17f867b327016462e4f2f84519c9643b62ac4"} err="failed to get container status \"927a14a66db06c1f045de17016d17f867b327016462e4f2f84519c9643b62ac4\": rpc error: code = NotFound desc = could not find container \"927a14a66db06c1f045de17016d17f867b327016462e4f2f84519c9643b62ac4\": container with ID starting with 927a14a66db06c1f045de17016d17f867b327016462e4f2f84519c9643b62ac4 not found: ID does not exist" Oct 29 23:16:29.212729 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.212681 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght"] Oct 29 23:16:29.219976 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.219947 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-zmght"] Oct 29 23:16:29.222140 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.222110 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:29.222365 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.222144 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:29.222365 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.222158 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m2h7\" (UniqueName: \"kubernetes.io/projected/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-kube-api-access-6m2h7\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:29.222365 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.222172 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:29.222365 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.222186 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:29.222365 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.222218 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:29.222365 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.222232 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6f4fd46-5aab-49a8-9446-f71ed103d2eb-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:29.222365 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.222244 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c0101f-490c-470e-b945-e3bb7c86d2c6-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:29.222365 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.222256 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctv5z\" (UniqueName: \"kubernetes.io/projected/62c0101f-490c-470e-b945-e3bb7c86d2c6-kube-api-access-ctv5z\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:29.252136 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.252096 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s"] Oct 29 23:16:29.259275 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.259251 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-frm5s"] Oct 29 23:16:29.451858 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.451767 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp"] Oct 29 23:16:29.459049 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.459012 4821 cpu_manager.go:414] "containerMap: removing container" podUID="e02c68ae-98e8-47ce-8b21-9aa5025a20d5" containerName="istio-proxy" Oct 29 23:16:29.459291 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.459274 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e02c68ae-98e8-47ce-8b21-9aa5025a20d5" containerName="istio-proxy" Oct 29 23:16:29.468323 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.468292 4821 cpu_manager.go:414] "containerMap: removing container" podUID="2aeb91d0-8e92-4c73-ae62-767cdb2e6f84" containerName="main" Oct 29 23:16:29.468598 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.468573 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aeb91d0-8e92-4c73-ae62-767cdb2e6f84" containerName="main" Oct 29 23:16:29.468749 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.468733 4821 cpu_manager.go:414] "containerMap: removing container" podUID="e6f4fd46-5aab-49a8-9446-f71ed103d2eb" containerName="istio-proxy" Oct 29 23:16:29.468878 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.468861 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6f4fd46-5aab-49a8-9446-f71ed103d2eb" containerName="istio-proxy" Oct 29 23:16:29.469027 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.469014 4821 cpu_manager.go:414] "containerMap: removing container" podUID="62c0101f-490c-470e-b945-e3bb7c86d2c6" containerName="istio-proxy" Oct 29 23:16:29.469120 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.469109 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c0101f-490c-470e-b945-e3bb7c86d2c6" containerName="istio-proxy" Oct 29 23:16:29.469231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.469219 4821 cpu_manager.go:414] "containerMap: removing container" podUID="041cb60f-5e7c-47d7-8e83-cce35d4c9529" containerName="istio-proxy" Oct 29 23:16:29.469327 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.469317 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="041cb60f-5e7c-47d7-8e83-cce35d4c9529" containerName="istio-proxy" Oct 29 23:16:29.469624 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.469609 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c0101f-490c-470e-b945-e3bb7c86d2c6" containerName="istio-proxy" Oct 29 23:16:29.473222 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.469724 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="041cb60f-5e7c-47d7-8e83-cce35d4c9529" containerName="istio-proxy" Oct 29 23:16:29.473406 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.473390 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="e02c68ae-98e8-47ce-8b21-9aa5025a20d5" containerName="istio-proxy" Oct 29 23:16:29.473493 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.473485 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aeb91d0-8e92-4c73-ae62-767cdb2e6f84" containerName="main" Oct 29 23:16:29.473578 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.473569 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6f4fd46-5aab-49a8-9446-f71ed103d2eb" containerName="istio-proxy" Oct 29 23:16:29.518602 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.510311 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.533525 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.532377 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-ea8e77-openshift-default-dockercfg-snwc2\"" Oct 29 23:16:29.537591 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.537556 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp"] Oct 29 23:16:29.643239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.640323 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-workload-socket\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.643239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.640380 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-token\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.643239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.640558 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-data\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.643239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.640620 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-workload-certs\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.643239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.640652 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-podinfo\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.643239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.640690 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b144b55c-1c80-469b-ba68-671fdc2e879f-istiod-ca-cert\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.643239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.640754 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-credential-socket\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.643239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.640849 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqhpf\" (UniqueName: \"kubernetes.io/projected/b144b55c-1c80-469b-ba68-671fdc2e879f-kube-api-access-xqhpf\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.643239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.640921 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-envoy\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.647814 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.647773 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn"] Oct 29 23:16:29.650497 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.650471 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-lfwsn"] Oct 29 23:16:29.687848 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.687816 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84"] Oct 29 23:16:29.692892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.692853 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-31226c-openshift-default-d5bcdb6c9-7gb84"] Oct 29 23:16:29.742504 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.742407 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-credential-socket\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.742504 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.742491 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqhpf\" (UniqueName: \"kubernetes.io/projected/b144b55c-1c80-469b-ba68-671fdc2e879f-kube-api-access-xqhpf\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.742743 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.742540 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-envoy\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.742743 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.742602 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-workload-socket\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.742743 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.742627 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-token\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.742743 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.742738 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-data\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.742960 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.742783 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-workload-certs\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.742960 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.742810 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-podinfo\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.742960 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.742847 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b144b55c-1c80-469b-ba68-671fdc2e879f-istiod-ca-cert\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.742960 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.742909 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-credential-socket\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.743177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.743047 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-workload-socket\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.743177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.743097 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-data\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.743177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.743163 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-workload-certs\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.743646 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.743623 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b144b55c-1c80-469b-ba68-671fdc2e879f-istiod-ca-cert\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.746249 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.746222 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-podinfo\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.748317 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.748291 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-envoy\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.768927 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.768616 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-token\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.770125 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.770095 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqhpf\" (UniqueName: \"kubernetes.io/projected/b144b55c-1c80-469b-ba68-671fdc2e879f-kube-api-access-xqhpf\") pod \"e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.862521 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.862483 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:29.878880 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:29.874841 4821 scope.go:122] "RemoveContainer" containerID="ca4df17ab31c988fd8078c36c196438dbbe5fadbb54a5f3ac166e5d3deb7bbaf" Oct 29 23:16:30.172086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:30.172049 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="041cb60f-5e7c-47d7-8e83-cce35d4c9529" path="/var/lib/kubelet/pods/041cb60f-5e7c-47d7-8e83-cce35d4c9529/volumes" Oct 29 23:16:30.172658 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:30.172641 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c0101f-490c-470e-b945-e3bb7c86d2c6" path="/var/lib/kubelet/pods/62c0101f-490c-470e-b945-e3bb7c86d2c6/volumes" Oct 29 23:16:30.173106 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:30.173093 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e02c68ae-98e8-47ce-8b21-9aa5025a20d5" path="/var/lib/kubelet/pods/e02c68ae-98e8-47ce-8b21-9aa5025a20d5/volumes" Oct 29 23:16:30.173917 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:30.173898 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6f4fd46-5aab-49a8-9446-f71ed103d2eb" path="/var/lib/kubelet/pods/e6f4fd46-5aab-49a8-9446-f71ed103d2eb/volumes" Oct 29 23:16:30.270931 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:30.270886 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:16:30.271071 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:30.270981 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:16:30.271071 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:30.271015 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:16:30.272026 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:30.271996 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp"] Oct 29 23:16:31.128553 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:31.128510 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" event={"ID":"b144b55c-1c80-469b-ba68-671fdc2e879f","Type":"ContainerStarted","Data":"3bf5211ecf3874f40e0fd0b7b56a7633f072a75a8cc6d15a5eda2ce46798f0d1"} Oct 29 23:16:31.128776 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:31.128584 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" event={"ID":"b144b55c-1c80-469b-ba68-671fdc2e879f","Type":"ContainerStarted","Data":"4e80bc4e31db75a0105e4de6f86f058f4d542ea127848e32a10a19246aa8f501"} Oct 29 23:16:31.190355 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:31.190304 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" podStartSLOduration=2.1902890839999998 podStartE2EDuration="2.190289084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:31.188793048 +0000 UTC m=+2045.129340957" watchObservedRunningTime="2026-10-29 23:16:31.190289084 +0000 UTC m=+2045.130836994" Oct 29 23:16:31.862818 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:31.862784 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:31.869113 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:31.869087 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:32.137985 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:32.137902 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:32.139092 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:32.139065 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:16:36.818405 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.817733 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl"] Oct 29 23:16:36.823662 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.823629 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.865594 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.865561 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f4ce70-openshift-default-dockercfg-qr8z4\"" Oct 29 23:16:36.873656 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.873621 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl"] Oct 29 23:16:36.875610 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.875577 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-data\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.875769 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.875655 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm77h\" (UniqueName: \"kubernetes.io/projected/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-kube-api-access-rm77h\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.875840 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.875819 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-credential-socket\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.875926 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.875864 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istiod-ca-cert\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.875984 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.875933 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-token\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.876068 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.876048 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-workload-certs\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.880218 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.876347 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-workload-socket\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.880218 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.876391 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-envoy\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.880218 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.876472 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-podinfo\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.978404 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.978362 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-data\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.978608 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.978444 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm77h\" (UniqueName: \"kubernetes.io/projected/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-kube-api-access-rm77h\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.978608 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.978557 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-credential-socket\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.978740 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.978632 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istiod-ca-cert\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.978825 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.978799 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-data\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.978908 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.978885 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-credential-socket\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.979385 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.979363 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istiod-ca-cert\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.979508 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.979490 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-token\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.979603 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.979592 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-workload-certs\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.979637 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.979621 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-workload-socket\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.979677 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.979642 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-envoy\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.979720 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.979680 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-podinfo\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.979958 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.979936 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-workload-certs\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.979958 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.979949 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-workload-socket\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.982244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.982217 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-envoy\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:36.982431 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:36.982409 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-podinfo\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:37.037255 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:37.036951 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm77h\" (UniqueName: \"kubernetes.io/projected/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-kube-api-access-rm77h\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:37.057464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:37.057435 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-token\") pod \"e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:37.155230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:37.154508 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:37.652566 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:37.652518 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:16:37.652727 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:37.652667 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:16:37.652803 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:37.652758 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:16:37.654861 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:37.654834 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl"] Oct 29 23:16:38.312349 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:38.312295 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" podUID="5ef833d5-5335-4391-b2cf-2dd970f88eed" containerName="main" containerID="cri-o://691bac9ba5048c3512f1c56c23595eb8318afe7f83e7788782ce9ae4d4ef23b6" gracePeriod=45 Oct 29 23:16:38.339995 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:38.339960 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" event={"ID":"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe","Type":"ContainerStarted","Data":"54363f00966d104348ac4181ee49432d4a9e9f1785ac761321df420926789766"} Oct 29 23:16:38.340120 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:38.340025 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" event={"ID":"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe","Type":"ContainerStarted","Data":"88a9ef8a0775ae8f976f07651cacee1c92ab8a43a1e1ec6d3568edaabe0b534a"} Oct 29 23:16:38.415305 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:38.415240 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" podStartSLOduration=2.41522482 podStartE2EDuration="2.41522482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:38.400550229 +0000 UTC m=+2052.341098138" watchObservedRunningTime="2026-10-29 23:16:38.41522482 +0000 UTC m=+2052.355772727" Oct 29 23:16:38.719071 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:38.719041 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:16:38.792500 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:38.792468 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-tmp-dir\") pod \"5ef833d5-5335-4391-b2cf-2dd970f88eed\" (UID: \"5ef833d5-5335-4391-b2cf-2dd970f88eed\") " Oct 29 23:16:38.792680 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:38.792512 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef833d5-5335-4391-b2cf-2dd970f88eed-kube-api-access-59v4v\" (UniqueName: \"kubernetes.io/projected/5ef833d5-5335-4391-b2cf-2dd970f88eed-kube-api-access-59v4v\") pod \"5ef833d5-5335-4391-b2cf-2dd970f88eed\" (UID: \"5ef833d5-5335-4391-b2cf-2dd970f88eed\") " Oct 29 23:16:38.792680 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:38.792601 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ef833d5-5335-4391-b2cf-2dd970f88eed-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ef833d5-5335-4391-b2cf-2dd970f88eed-tls-certs\") pod \"5ef833d5-5335-4391-b2cf-2dd970f88eed\" (UID: \"5ef833d5-5335-4391-b2cf-2dd970f88eed\") " Oct 29 23:16:38.792828 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:38.792765 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-tmp-dir" pod "5ef833d5-5335-4391-b2cf-2dd970f88eed" (UID: "5ef833d5-5335-4391-b2cf-2dd970f88eed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:38.792828 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:38.792770 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-model-cache\") pod \"5ef833d5-5335-4391-b2cf-2dd970f88eed\" (UID: \"5ef833d5-5335-4391-b2cf-2dd970f88eed\") " Oct 29 23:16:38.792949 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:38.792859 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-home\" (UniqueName: \"kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-home\") pod \"5ef833d5-5335-4391-b2cf-2dd970f88eed\" (UID: \"5ef833d5-5335-4391-b2cf-2dd970f88eed\") " Oct 29 23:16:38.793015 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:38.792943 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-model-cache" pod "5ef833d5-5335-4391-b2cf-2dd970f88eed" (UID: "5ef833d5-5335-4391-b2cf-2dd970f88eed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:38.793080 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:38.793008 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-dshm\") pod \"5ef833d5-5335-4391-b2cf-2dd970f88eed\" (UID: \"5ef833d5-5335-4391-b2cf-2dd970f88eed\") " Oct 29 23:16:38.793450 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:38.793419 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-home" pod "5ef833d5-5335-4391-b2cf-2dd970f88eed" (UID: "5ef833d5-5335-4391-b2cf-2dd970f88eed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:38.793988 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:38.793964 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:38.794134 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:38.794118 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:38.794259 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:38.794247 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:38.796901 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:38.796878 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef833d5-5335-4391-b2cf-2dd970f88eed-kube-api-access-59v4v" pod "5ef833d5-5335-4391-b2cf-2dd970f88eed" (UID: "5ef833d5-5335-4391-b2cf-2dd970f88eed"). InnerVolumeSpecName "kube-api-access-59v4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:16:38.801654 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:38.801623 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-dshm" pod "5ef833d5-5335-4391-b2cf-2dd970f88eed" (UID: "5ef833d5-5335-4391-b2cf-2dd970f88eed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:38.801654 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:38.801625 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ef833d5-5335-4391-b2cf-2dd970f88eed-tls-certs" pod "5ef833d5-5335-4391-b2cf-2dd970f88eed" (UID: "5ef833d5-5335-4391-b2cf-2dd970f88eed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:16:38.896002 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:38.895963 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59v4v\" (UniqueName: \"kubernetes.io/projected/5ef833d5-5335-4391-b2cf-2dd970f88eed-kube-api-access-59v4v\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:38.896002 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:38.896000 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ef833d5-5335-4391-b2cf-2dd970f88eed-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:38.896283 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:38.896017 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ef833d5-5335-4391-b2cf-2dd970f88eed-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:39.155577 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:39.155544 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:39.162570 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:39.162536 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:39.348850 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:39.348817 4821 generic.go:356] "Generic (PLEG): container finished" podID="5ef833d5-5335-4391-b2cf-2dd970f88eed" containerID="691bac9ba5048c3512f1c56c23595eb8318afe7f83e7788782ce9ae4d4ef23b6" exitCode=0 Oct 29 23:16:39.349310 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:39.348857 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" Oct 29 23:16:39.349310 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:39.348891 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" event={"ID":"5ef833d5-5335-4391-b2cf-2dd970f88eed","Type":"ContainerDied","Data":"691bac9ba5048c3512f1c56c23595eb8318afe7f83e7788782ce9ae4d4ef23b6"} Oct 29 23:16:39.349310 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:39.348933 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" event={"ID":"5ef833d5-5335-4391-b2cf-2dd970f88eed","Type":"ContainerDied","Data":"c88ef88d37b7a3eef02204829109cbcb783fad35fad6c5c8f97583c6283b00d1"} Oct 29 23:16:39.349310 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:39.348951 4821 scope.go:122] "RemoveContainer" containerID="691bac9ba5048c3512f1c56c23595eb8318afe7f83e7788782ce9ae4d4ef23b6" Oct 29 23:16:39.349310 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:39.349238 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:39.350819 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:39.350797 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:16:39.360595 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:39.360573 4821 scope.go:122] "RemoveContainer" containerID="691bac9ba5048c3512f1c56c23595eb8318afe7f83e7788782ce9ae4d4ef23b6" Oct 29 23:16:39.360858 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:16:39.360840 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"691bac9ba5048c3512f1c56c23595eb8318afe7f83e7788782ce9ae4d4ef23b6\": container with ID starting with 691bac9ba5048c3512f1c56c23595eb8318afe7f83e7788782ce9ae4d4ef23b6 not found: ID does not exist" containerID="691bac9ba5048c3512f1c56c23595eb8318afe7f83e7788782ce9ae4d4ef23b6" Oct 29 23:16:39.360905 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:39.360867 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"691bac9ba5048c3512f1c56c23595eb8318afe7f83e7788782ce9ae4d4ef23b6"} err="failed to get container status \"691bac9ba5048c3512f1c56c23595eb8318afe7f83e7788782ce9ae4d4ef23b6\": rpc error: code = NotFound desc = could not find container \"691bac9ba5048c3512f1c56c23595eb8318afe7f83e7788782ce9ae4d4ef23b6\": container with ID starting with 691bac9ba5048c3512f1c56c23595eb8318afe7f83e7788782ce9ae4d4ef23b6 not found: ID does not exist" Oct 29 23:16:39.406674 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:39.406577 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7"] Oct 29 23:16:39.411768 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:39.411739 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7"] Oct 29 23:16:39.705319 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:39.705223 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-b0c3c3/auth-test-model-b0c3c3-kserve-8677db688f-w5nk7" podUID="5ef833d5-5335-4391-b2cf-2dd970f88eed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.205:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:16:40.171129 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:40.171084 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ef833d5-5335-4391-b2cf-2dd970f88eed" path="/var/lib/kubelet/pods/5ef833d5-5335-4391-b2cf-2dd970f88eed/volumes" Oct 29 23:16:41.421543 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:41.421513 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z"] Oct 29 23:16:41.422247 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:41.422232 4821 cpu_manager.go:414] "containerMap: removing container" podUID="5ef833d5-5335-4391-b2cf-2dd970f88eed" containerName="main" Oct 29 23:16:41.422313 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:41.422248 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef833d5-5335-4391-b2cf-2dd970f88eed" containerName="main" Oct 29 23:16:41.422368 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:41.422358 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef833d5-5335-4391-b2cf-2dd970f88eed" containerName="main" Oct 29 23:16:41.427470 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:41.427446 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z" Oct 29 23:16:41.454254 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:41.454226 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z"] Oct 29 23:16:41.460274 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:41.460251 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-e170ed-serving-cert\"" Oct 29 23:16:41.538356 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:41.538318 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd770541-8afc-468c-bdcb-f95ec581d689-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z\" (UID: \"bd770541-8afc-468c-bdcb-f95ec581d689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z" Oct 29 23:16:41.538589 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:41.538560 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd770541-8afc-468c-bdcb-f95ec581d689-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z\" (UID: \"bd770541-8afc-468c-bdcb-f95ec581d689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z" Oct 29 23:16:41.538794 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:41.538774 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64xrs\" (UniqueName: \"kubernetes.io/projected/bd770541-8afc-468c-bdcb-f95ec581d689-kube-api-access-64xrs\") pod \"maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z\" (UID: \"bd770541-8afc-468c-bdcb-f95ec581d689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z" Oct 29 23:16:41.538897 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:41.538819 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd770541-8afc-468c-bdcb-f95ec581d689-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z\" (UID: \"bd770541-8afc-468c-bdcb-f95ec581d689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z" Oct 29 23:16:41.655225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:41.654234 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd770541-8afc-468c-bdcb-f95ec581d689-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z\" (UID: \"bd770541-8afc-468c-bdcb-f95ec581d689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z" Oct 29 23:16:41.655225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:41.654424 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64xrs\" (UniqueName: \"kubernetes.io/projected/bd770541-8afc-468c-bdcb-f95ec581d689-kube-api-access-64xrs\") pod \"maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z\" (UID: \"bd770541-8afc-468c-bdcb-f95ec581d689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z" Oct 29 23:16:41.655225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:41.654468 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd770541-8afc-468c-bdcb-f95ec581d689-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z\" (UID: \"bd770541-8afc-468c-bdcb-f95ec581d689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z" Oct 29 23:16:41.655225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:41.654607 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd770541-8afc-468c-bdcb-f95ec581d689-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z\" (UID: \"bd770541-8afc-468c-bdcb-f95ec581d689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z" Oct 29 23:16:41.661593 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:41.655809 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd770541-8afc-468c-bdcb-f95ec581d689-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z\" (UID: \"bd770541-8afc-468c-bdcb-f95ec581d689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z" Oct 29 23:16:41.671875 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:41.671793 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd770541-8afc-468c-bdcb-f95ec581d689-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z\" (UID: \"bd770541-8afc-468c-bdcb-f95ec581d689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z" Oct 29 23:16:41.680483 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:41.680372 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd770541-8afc-468c-bdcb-f95ec581d689-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z\" (UID: \"bd770541-8afc-468c-bdcb-f95ec581d689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z" Oct 29 23:16:41.694177 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:41.694149 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64xrs\" (UniqueName: \"kubernetes.io/projected/bd770541-8afc-468c-bdcb-f95ec581d689-kube-api-access-64xrs\") pod \"maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z\" (UID: \"bd770541-8afc-468c-bdcb-f95ec581d689\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z" Oct 29 23:16:41.743173 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:41.743133 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z" Oct 29 23:16:42.068007 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:16:42.067975 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd770541_8afc_468c_bdcb_f95ec581d689.slice/crio-d436533c58141e4f20e7e713942decc44a20272f25ff96562314d22ce7bfd390 WatchSource:0}: Error finding container d436533c58141e4f20e7e713942decc44a20272f25ff96562314d22ce7bfd390: Status 404 returned error can't find the container with id d436533c58141e4f20e7e713942decc44a20272f25ff96562314d22ce7bfd390 Oct 29 23:16:42.068842 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:42.068818 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z"] Oct 29 23:16:42.383686 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:42.383643 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z" event={"ID":"bd770541-8afc-468c-bdcb-f95ec581d689","Type":"ContainerStarted","Data":"d436533c58141e4f20e7e713942decc44a20272f25ff96562314d22ce7bfd390"} Oct 29 23:16:42.570958 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:42.570911 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8"] Oct 29 23:16:42.576828 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:42.576804 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8" Oct 29 23:16:42.596447 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:42.596412 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f4ce70\"" Oct 29 23:16:42.601365 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:42.601305 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8"] Oct 29 23:16:42.647989 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:42.647901 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8"] Oct 29 23:16:42.653191 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:42.653168 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8" Oct 29 23:16:42.672223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:42.671399 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f4ce70-dockercfg-gr92z\"" Oct 29 23:16:42.677856 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:42.676588 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fz6m\" (UniqueName: \"kubernetes.io/projected/60b4edf6-47dc-4829-bdaf-78c062cdce78-kube-api-access-6fz6m\") pod \"payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8\" (UID: \"60b4edf6-47dc-4829-bdaf-78c062cdce78\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8" Oct 29 23:16:42.677856 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:42.676771 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b4edf6-47dc-4829-bdaf-78c062cdce78-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8\" (UID: \"60b4edf6-47dc-4829-bdaf-78c062cdce78\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8" Oct 29 23:16:42.677856 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:42.677274 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8"] Oct 29 23:16:42.781222 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:42.781157 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fz6m\" (UniqueName: \"kubernetes.io/projected/60b4edf6-47dc-4829-bdaf-78c062cdce78-kube-api-access-6fz6m\") pod \"payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8\" (UID: \"60b4edf6-47dc-4829-bdaf-78c062cdce78\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8" Oct 29 23:16:42.782019 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:42.781988 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9hkb\" (UniqueName: \"kubernetes.io/projected/ddb7ef9a-3948-4f03-ab50-3a9629a3c67d-kube-api-access-f9hkb\") pod \"payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8\" (UID: \"ddb7ef9a-3948-4f03-ab50-3a9629a3c67d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8" Oct 29 23:16:42.782688 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:42.782665 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb7ef9a-3948-4f03-ab50-3a9629a3c67d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8\" (UID: \"ddb7ef9a-3948-4f03-ab50-3a9629a3c67d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8" Oct 29 23:16:42.782905 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:42.782888 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b4edf6-47dc-4829-bdaf-78c062cdce78-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8\" (UID: \"60b4edf6-47dc-4829-bdaf-78c062cdce78\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8" Oct 29 23:16:42.784255 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:42.783971 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b4edf6-47dc-4829-bdaf-78c062cdce78-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8\" (UID: \"60b4edf6-47dc-4829-bdaf-78c062cdce78\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8" Oct 29 23:16:42.835265 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:42.817106 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fz6m\" (UniqueName: \"kubernetes.io/projected/60b4edf6-47dc-4829-bdaf-78c062cdce78-kube-api-access-6fz6m\") pod \"payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8\" (UID: \"60b4edf6-47dc-4829-bdaf-78c062cdce78\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8" Oct 29 23:16:42.911286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:42.889428 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9hkb\" (UniqueName: \"kubernetes.io/projected/ddb7ef9a-3948-4f03-ab50-3a9629a3c67d-kube-api-access-f9hkb\") pod \"payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8\" (UID: \"ddb7ef9a-3948-4f03-ab50-3a9629a3c67d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8" Oct 29 23:16:42.911286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:42.889491 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb7ef9a-3948-4f03-ab50-3a9629a3c67d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8\" (UID: \"ddb7ef9a-3948-4f03-ab50-3a9629a3c67d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8" Oct 29 23:16:42.911286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:42.890784 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb7ef9a-3948-4f03-ab50-3a9629a3c67d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8\" (UID: \"ddb7ef9a-3948-4f03-ab50-3a9629a3c67d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8" Oct 29 23:16:43.006536 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:42.971581 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8" Oct 29 23:16:43.006536 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.006366 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9hkb\" (UniqueName: \"kubernetes.io/projected/ddb7ef9a-3948-4f03-ab50-3a9629a3c67d-kube-api-access-f9hkb\") pod \"payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8\" (UID: \"ddb7ef9a-3948-4f03-ab50-3a9629a3c67d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8" Oct 29 23:16:43.026531 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.019767 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8" Oct 29 23:16:43.282483 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.282439 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" podUID="89e6c6b4-ed1a-4f8a-985f-238f0116574a" containerName="main" containerID="cri-o://a6f38b54e994a0d6309eba5537dd87a72339946e0e2d5e109c202e18096e9cb0" gracePeriod=45 Oct 29 23:16:43.401609 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.401568 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z" event={"ID":"bd770541-8afc-468c-bdcb-f95ec581d689","Type":"ContainerStarted","Data":"1b08cd8a594a96ba7a9ecd1f7dec8b2dfda8a0475d55d58ab769c16357334539"} Oct 29 23:16:43.402692 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.402668 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z" Oct 29 23:16:43.477969 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.477899 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z" podStartSLOduration=2.156482315 podStartE2EDuration="2.4778792s" totalImagesPullingTime="321.396885ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:43.471557373 +0000 UTC m=+2057.412105283" watchObservedRunningTime="2026-10-29 23:16:43.4778792 +0000 UTC m=+2057.418427111" Oct 29 23:16:43.533753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.533719 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8"] Oct 29 23:16:43.556793 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.556768 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:16:43.561755 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.561730 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8"] Oct 29 23:16:43.576351 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:16:43.576326 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b4edf6_47dc_4829_bdaf_78c062cdce78.slice/crio-e4bd78eb169a6da1f9ede589f763812282103157b83a5ac2adc60c77cbaecc28 WatchSource:0}: Error finding container e4bd78eb169a6da1f9ede589f763812282103157b83a5ac2adc60c77cbaecc28: Status 404 returned error can't find the container with id e4bd78eb169a6da1f9ede589f763812282103157b83a5ac2adc60c77cbaecc28 Oct 29 23:16:43.637348 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.637024 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89e6c6b4-ed1a-4f8a-985f-238f0116574a-tls-certs\" (UniqueName: \"kubernetes.io/secret/89e6c6b4-ed1a-4f8a-985f-238f0116574a-tls-certs\") pod \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\" (UID: \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\") " Oct 29 23:16:43.637348 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.637147 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89e6c6b4-ed1a-4f8a-985f-238f0116574a-kube-api-access-pqjhm\" (UniqueName: \"kubernetes.io/projected/89e6c6b4-ed1a-4f8a-985f-238f0116574a-kube-api-access-pqjhm\") pod \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\" (UID: \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\") " Oct 29 23:16:43.637348 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.637294 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-tmp-dir\") pod \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\" (UID: \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\") " Oct 29 23:16:43.637348 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.637332 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-model-cache\") pod \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\" (UID: \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\") " Oct 29 23:16:43.637840 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.637386 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-home\" (UniqueName: \"kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-home\") pod \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\" (UID: \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\") " Oct 29 23:16:43.637840 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.637617 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-dshm\") pod \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\" (UID: \"89e6c6b4-ed1a-4f8a-985f-238f0116574a\") " Oct 29 23:16:43.638769 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.638367 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-tmp-dir" pod "89e6c6b4-ed1a-4f8a-985f-238f0116574a" (UID: "89e6c6b4-ed1a-4f8a-985f-238f0116574a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:43.638769 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.638570 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-model-cache" pod "89e6c6b4-ed1a-4f8a-985f-238f0116574a" (UID: "89e6c6b4-ed1a-4f8a-985f-238f0116574a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:43.638769 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.638735 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-home" pod "89e6c6b4-ed1a-4f8a-985f-238f0116574a" (UID: "89e6c6b4-ed1a-4f8a-985f-238f0116574a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:43.639228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.639188 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:43.639228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.639228 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:43.639566 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.639236 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:43.644284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.644250 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89e6c6b4-ed1a-4f8a-985f-238f0116574a-tls-certs" pod "89e6c6b4-ed1a-4f8a-985f-238f0116574a" (UID: "89e6c6b4-ed1a-4f8a-985f-238f0116574a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:16:43.648102 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.648061 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-dshm" pod "89e6c6b4-ed1a-4f8a-985f-238f0116574a" (UID: "89e6c6b4-ed1a-4f8a-985f-238f0116574a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:16:43.667860 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.667824 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e6c6b4-ed1a-4f8a-985f-238f0116574a-kube-api-access-pqjhm" pod "89e6c6b4-ed1a-4f8a-985f-238f0116574a" (UID: "89e6c6b4-ed1a-4f8a-985f-238f0116574a"). InnerVolumeSpecName "kube-api-access-pqjhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:16:43.765329 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.765294 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89e6c6b4-ed1a-4f8a-985f-238f0116574a-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:43.765686 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.765667 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89e6c6b4-ed1a-4f8a-985f-238f0116574a-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:43.765800 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:43.765789 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqjhm\" (UniqueName: \"kubernetes.io/projected/89e6c6b4-ed1a-4f8a-985f-238f0116574a-kube-api-access-pqjhm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:16:44.411128 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:44.411095 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8" event={"ID":"60b4edf6-47dc-4829-bdaf-78c062cdce78","Type":"ContainerStarted","Data":"ae00281d351a4ae41ec596da95b36eaa3ad4cb063f858a6a9231d0608296e816"} Oct 29 23:16:44.411351 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:44.411149 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8" event={"ID":"60b4edf6-47dc-4829-bdaf-78c062cdce78","Type":"ContainerStarted","Data":"e4bd78eb169a6da1f9ede589f763812282103157b83a5ac2adc60c77cbaecc28"} Oct 29 23:16:44.411351 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:44.411188 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8" Oct 29 23:16:44.412706 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:44.412675 4821 generic.go:356] "Generic (PLEG): container finished" podID="89e6c6b4-ed1a-4f8a-985f-238f0116574a" containerID="a6f38b54e994a0d6309eba5537dd87a72339946e0e2d5e109c202e18096e9cb0" exitCode=0 Oct 29 23:16:44.412836 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:44.412715 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" Oct 29 23:16:44.412836 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:44.412756 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" event={"ID":"89e6c6b4-ed1a-4f8a-985f-238f0116574a","Type":"ContainerDied","Data":"a6f38b54e994a0d6309eba5537dd87a72339946e0e2d5e109c202e18096e9cb0"} Oct 29 23:16:44.412836 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:44.412793 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp" event={"ID":"89e6c6b4-ed1a-4f8a-985f-238f0116574a","Type":"ContainerDied","Data":"d7072dbc980f054cc7d9b17aa268d0eb243e968419b5b90711b0c8bbeb791894"} Oct 29 23:16:44.412836 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:44.412811 4821 scope.go:122] "RemoveContainer" containerID="a6f38b54e994a0d6309eba5537dd87a72339946e0e2d5e109c202e18096e9cb0" Oct 29 23:16:44.414751 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:44.414729 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8" event={"ID":"ddb7ef9a-3948-4f03-ab50-3a9629a3c67d","Type":"ContainerStarted","Data":"f400b0bc8b7a4e887ba09dd4026101407040fa498d529e4e259b2c70384312cf"} Oct 29 23:16:44.414869 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:44.414779 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8" event={"ID":"ddb7ef9a-3948-4f03-ab50-3a9629a3c67d","Type":"ContainerStarted","Data":"3133e2a438319d7b0fd07dbfce8f4711dc3642fab239b2c3451721ea47f475e2"} Oct 29 23:16:44.414956 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:44.414901 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8" Oct 29 23:16:44.423674 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:44.423656 4821 scope.go:122] "RemoveContainer" containerID="a6f38b54e994a0d6309eba5537dd87a72339946e0e2d5e109c202e18096e9cb0" Oct 29 23:16:44.423957 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:16:44.423928 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6f38b54e994a0d6309eba5537dd87a72339946e0e2d5e109c202e18096e9cb0\": container with ID starting with a6f38b54e994a0d6309eba5537dd87a72339946e0e2d5e109c202e18096e9cb0 not found: ID does not exist" containerID="a6f38b54e994a0d6309eba5537dd87a72339946e0e2d5e109c202e18096e9cb0" Oct 29 23:16:44.424038 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:44.423954 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f38b54e994a0d6309eba5537dd87a72339946e0e2d5e109c202e18096e9cb0"} err="failed to get container status \"a6f38b54e994a0d6309eba5537dd87a72339946e0e2d5e109c202e18096e9cb0\": rpc error: code = NotFound desc = could not find container \"a6f38b54e994a0d6309eba5537dd87a72339946e0e2d5e109c202e18096e9cb0\": container with ID starting with a6f38b54e994a0d6309eba5537dd87a72339946e0e2d5e109c202e18096e9cb0 not found: ID does not exist" Oct 29 23:16:44.458446 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:44.458361 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8" podStartSLOduration=2.458348526 podStartE2EDuration="2.458348526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:44.455560849 +0000 UTC m=+2058.396108759" watchObservedRunningTime="2026-10-29 23:16:44.458348526 +0000 UTC m=+2058.398896436" Oct 29 23:16:44.501669 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:44.501623 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8" podStartSLOduration=2.501609396 podStartE2EDuration="2.501609396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:44.497510139 +0000 UTC m=+2058.438058051" watchObservedRunningTime="2026-10-29 23:16:44.501609396 +0000 UTC m=+2058.442157300" Oct 29 23:16:44.538725 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:44.538694 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp"] Oct 29 23:16:44.544775 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:44.544743 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ed3bf2/auth-test-model-ed3bf2-kserve-d9564c669-qbftp"] Oct 29 23:16:45.330260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:45.330221 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98"] Oct 29 23:16:45.330950 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:45.330934 4821 cpu_manager.go:414] "containerMap: removing container" podUID="89e6c6b4-ed1a-4f8a-985f-238f0116574a" containerName="main" Oct 29 23:16:45.331004 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:45.330953 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e6c6b4-ed1a-4f8a-985f-238f0116574a" containerName="main" Oct 29 23:16:45.331084 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:45.331075 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="89e6c6b4-ed1a-4f8a-985f-238f0116574a" containerName="main" Oct 29 23:16:45.335919 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:45.335898 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98" Oct 29 23:16:45.357562 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:45.357537 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98"] Oct 29 23:16:45.367282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:45.361666 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-9e7cc9-serving-cert\"" Oct 29 23:16:45.387363 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:45.387333 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9c30d41-405c-49ee-931c-66f58edcd0dd-maas-api-tls\") pod \"maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98\" (UID: \"e9c30d41-405c-49ee-931c-66f58edcd0dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98" Oct 29 23:16:45.387518 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:45.387480 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c30d41-405c-49ee-931c-66f58edcd0dd-metrics-certs\") pod \"maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98\" (UID: \"e9c30d41-405c-49ee-931c-66f58edcd0dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98" Oct 29 23:16:45.387686 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:45.387662 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbxz2\" (UniqueName: \"kubernetes.io/projected/e9c30d41-405c-49ee-931c-66f58edcd0dd-kube-api-access-cbxz2\") pod \"maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98\" (UID: \"e9c30d41-405c-49ee-931c-66f58edcd0dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98" Oct 29 23:16:45.387793 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:45.387776 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9c30d41-405c-49ee-931c-66f58edcd0dd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98\" (UID: \"e9c30d41-405c-49ee-931c-66f58edcd0dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98" Oct 29 23:16:45.489555 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:45.489522 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c30d41-405c-49ee-931c-66f58edcd0dd-metrics-certs\") pod \"maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98\" (UID: \"e9c30d41-405c-49ee-931c-66f58edcd0dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98" Oct 29 23:16:45.489738 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:45.489691 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbxz2\" (UniqueName: \"kubernetes.io/projected/e9c30d41-405c-49ee-931c-66f58edcd0dd-kube-api-access-cbxz2\") pod \"maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98\" (UID: \"e9c30d41-405c-49ee-931c-66f58edcd0dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98" Oct 29 23:16:45.490286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:45.490252 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9c30d41-405c-49ee-931c-66f58edcd0dd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98\" (UID: \"e9c30d41-405c-49ee-931c-66f58edcd0dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98" Oct 29 23:16:45.490484 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:45.490457 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9c30d41-405c-49ee-931c-66f58edcd0dd-maas-api-tls\") pod \"maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98\" (UID: \"e9c30d41-405c-49ee-931c-66f58edcd0dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98" Oct 29 23:16:45.491149 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:45.491125 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9c30d41-405c-49ee-931c-66f58edcd0dd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98\" (UID: \"e9c30d41-405c-49ee-931c-66f58edcd0dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98" Oct 29 23:16:45.492730 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:45.492711 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c30d41-405c-49ee-931c-66f58edcd0dd-metrics-certs\") pod \"maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98\" (UID: \"e9c30d41-405c-49ee-931c-66f58edcd0dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98" Oct 29 23:16:45.493073 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:45.493050 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9c30d41-405c-49ee-931c-66f58edcd0dd-maas-api-tls\") pod \"maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98\" (UID: \"e9c30d41-405c-49ee-931c-66f58edcd0dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98" Oct 29 23:16:45.516546 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:45.516513 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbxz2\" (UniqueName: \"kubernetes.io/projected/e9c30d41-405c-49ee-931c-66f58edcd0dd-kube-api-access-cbxz2\") pod \"maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98\" (UID: \"e9c30d41-405c-49ee-931c-66f58edcd0dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98" Oct 29 23:16:45.650548 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:45.650511 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98" Oct 29 23:16:45.978564 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:45.978532 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98"] Oct 29 23:16:45.978704 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:16:45.978682 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c30d41_405c_49ee_931c_66f58edcd0dd.slice/crio-49f461df9f270beb3a204a55110d8fae5dcb6940c30a3396d3e3fd9a394d04fb WatchSource:0}: Error finding container 49f461df9f270beb3a204a55110d8fae5dcb6940c30a3396d3e3fd9a394d04fb: Status 404 returned error can't find the container with id 49f461df9f270beb3a204a55110d8fae5dcb6940c30a3396d3e3fd9a394d04fb Oct 29 23:16:46.172416 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:46.172380 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89e6c6b4-ed1a-4f8a-985f-238f0116574a" path="/var/lib/kubelet/pods/89e6c6b4-ed1a-4f8a-985f-238f0116574a/volumes" Oct 29 23:16:46.435452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:46.435413 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98" event={"ID":"e9c30d41-405c-49ee-931c-66f58edcd0dd","Type":"ContainerStarted","Data":"b9826eedda2fe7a734d00923ced1cbf70e355ad18bf7037c45e5292c8e59f372"} Oct 29 23:16:46.435841 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:46.435527 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98" event={"ID":"e9c30d41-405c-49ee-931c-66f58edcd0dd","Type":"ContainerStarted","Data":"49f461df9f270beb3a204a55110d8fae5dcb6940c30a3396d3e3fd9a394d04fb"} Oct 29 23:16:46.435841 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:46.435562 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98" Oct 29 23:16:50.430064 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:50.430027 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z" Oct 29 23:16:50.480120 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:50.480069 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98" podStartSLOduration=5.108504084 podStartE2EDuration="5.480055243s" totalImagesPullingTime="371.551159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:46.53729367 +0000 UTC m=+2060.477841561" watchObservedRunningTime="2026-10-29 23:16:50.480055243 +0000 UTC m=+2064.420603153" Oct 29 23:16:52.451653 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:52.451622 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98" Oct 29 23:16:53.150934 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.150900 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s"] Oct 29 23:16:53.155852 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.155820 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.172082 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.172058 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-b0c3c3-openshift-default-dockercfg-nzrxs\"" Oct 29 23:16:53.193256 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.193163 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s"] Oct 29 23:16:53.237739 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.237693 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rtbg\" (UniqueName: \"kubernetes.io/projected/3b626717-62b8-428e-8371-9244e9f3e55f-kube-api-access-7rtbg\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.237953 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.237796 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-workload-socket\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.237953 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.237897 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b626717-62b8-428e-8371-9244e9f3e55f-istio-podinfo\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.238075 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.237952 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-credential-socket\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.238159 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.238138 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-workload-certs\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.238566 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.238532 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-istio-data\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.238641 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.238575 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b626717-62b8-428e-8371-9244e9f3e55f-istio-token\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.238641 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.238602 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-istio-envoy\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.238641 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.238630 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b626717-62b8-428e-8371-9244e9f3e55f-istiod-ca-cert\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.340620 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.340441 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-istio-data\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.340620 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.340497 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b626717-62b8-428e-8371-9244e9f3e55f-istio-token\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.340620 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.340529 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-istio-envoy\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.340620 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.340559 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b626717-62b8-428e-8371-9244e9f3e55f-istiod-ca-cert\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.341063 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.340667 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rtbg\" (UniqueName: \"kubernetes.io/projected/3b626717-62b8-428e-8371-9244e9f3e55f-kube-api-access-7rtbg\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.341063 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.340724 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-workload-socket\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.341063 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.340816 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b626717-62b8-428e-8371-9244e9f3e55f-istio-podinfo\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.341063 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.340867 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-credential-socket\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.341063 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.341034 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-workload-certs\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.341565 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.341535 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-workload-certs\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.341877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.341842 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-istio-data\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.342715 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.342581 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-workload-socket\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.343128 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.343103 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b626717-62b8-428e-8371-9244e9f3e55f-istiod-ca-cert\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.343410 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.343392 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-credential-socket\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.363103 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.345992 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-istio-envoy\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.369039 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.364002 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b626717-62b8-428e-8371-9244e9f3e55f-istio-podinfo\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.399997 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.399961 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rtbg\" (UniqueName: \"kubernetes.io/projected/3b626717-62b8-428e-8371-9244e9f3e55f-kube-api-access-7rtbg\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.412270 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.409791 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b626717-62b8-428e-8371-9244e9f3e55f-istio-token\") pod \"e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.477228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:53.475547 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:53.906586 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:16:53.906502 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:16:54.074370 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:54.074083 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s"] Oct 29 23:16:54.081227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:54.081109 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:16:54.081399 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:54.081238 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:16:54.081399 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:54.081287 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:16:54.529282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:54.529245 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" event={"ID":"3b626717-62b8-428e-8371-9244e9f3e55f","Type":"ContainerStarted","Data":"5d7f9de66f2f49a52a155a04af7c19b03061860c1d58826c5a22722d011c2270"} Oct 29 23:16:54.529771 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:54.529300 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" event={"ID":"3b626717-62b8-428e-8371-9244e9f3e55f","Type":"ContainerStarted","Data":"699185642e75f92af35271176e80ca113ee90ae0fd77fcb31f3504d805f50eda"} Oct 29 23:16:54.594267 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:54.594138 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" podStartSLOduration=1.594117728 podStartE2EDuration="1.594117728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:54.576537023 +0000 UTC m=+2068.517084949" watchObservedRunningTime="2026-10-29 23:16:54.594117728 +0000 UTC m=+2068.534665638" Oct 29 23:16:55.047386 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:55.047116 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b85895da-llgbx"] Oct 29 23:16:55.053640 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:55.053610 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b85895da-llgbx"] Oct 29 23:16:55.445162 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:55.444810 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8" Oct 29 23:16:55.445396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:55.445235 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8" Oct 29 23:16:55.476725 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:55.476695 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:16:55.478605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:55.478579 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Oct 29 23:16:55.478748 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:55.478627 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" podUID="3b626717-62b8-428e-8371-9244e9f3e55f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Oct 29 23:16:56.173891 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.173854 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b496027f-7bfe-475d-be67-d70d3782952d" path="/var/lib/kubelet/pods/b496027f-7bfe-475d-be67-d70d3782952d/volumes" Oct 29 23:16:56.345396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.345359 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q"] Oct 29 23:16:56.350268 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.350250 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.370625 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.370597 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q"] Oct 29 23:16:56.377666 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.377638 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fba83e-openshift-default-dockercfg-s7cj7\"" Oct 29 23:16:56.473790 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.473698 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-token\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.473961 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.473889 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-data\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.473961 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.473940 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-envoy\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.474278 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.474195 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istiod-ca-cert\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.474353 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.474307 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br6x6\" (UniqueName: \"kubernetes.io/projected/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-kube-api-access-br6x6\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.474433 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.474411 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-credential-socket\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.474524 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.474510 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-workload-socket\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.474604 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.474587 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-podinfo\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.474656 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.474613 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-workload-certs\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.476536 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.476511 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Oct 29 23:16:56.476607 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.476572 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" podUID="3b626717-62b8-428e-8371-9244e9f3e55f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Oct 29 23:16:56.586808 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.586763 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-data\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.587733 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.587691 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-envoy\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.587887 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.587844 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istiod-ca-cert\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.587887 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.587850 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-data\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.587887 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.587878 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br6x6\" (UniqueName: \"kubernetes.io/projected/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-kube-api-access-br6x6\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.588666 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.588639 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-credential-socket\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.589126 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.589004 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-workload-socket\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.589263 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.589181 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-podinfo\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.589263 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.589231 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-workload-certs\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.589388 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.589281 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-token\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.589388 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.589311 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-credential-socket\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.589883 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.589858 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-workload-certs\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.589978 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.589879 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-workload-socket\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.592045 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.592015 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-envoy\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.592739 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.592046 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istiod-ca-cert\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.592739 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.592491 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-podinfo\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.612877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.612852 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-token\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.612984 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.612888 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br6x6\" (UniqueName: \"kubernetes.io/projected/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-kube-api-access-br6x6\") pod \"e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:56.665687 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:56.665652 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:57.064895 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:16:57.064860 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bcf37ba_4f91_43c4_ac2d_2cc310f9b7ad.slice/crio-a8ba125434395b671461b77b8cac1517c17fcf265b67104e3f9e4fe695a08fcc WatchSource:0}: Error finding container a8ba125434395b671461b77b8cac1517c17fcf265b67104e3f9e4fe695a08fcc: Status 404 returned error can't find the container with id a8ba125434395b671461b77b8cac1517c17fcf265b67104e3f9e4fe695a08fcc Oct 29 23:16:57.066857 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:57.066827 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q"] Oct 29 23:16:57.069552 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:57.069515 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:16:57.069672 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:57.069600 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:16:57.069672 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:57.069648 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:16:57.476901 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:57.476867 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Oct 29 23:16:57.477343 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:57.476919 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" podUID="3b626717-62b8-428e-8371-9244e9f3e55f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Oct 29 23:16:57.605479 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:57.605438 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" event={"ID":"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad","Type":"ContainerStarted","Data":"6c6436c8e2d277135120be699fe00780fc1a1f8d03ec0dd25614743cc6f73893"} Oct 29 23:16:57.605733 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:57.605708 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" event={"ID":"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad","Type":"ContainerStarted","Data":"a8ba125434395b671461b77b8cac1517c17fcf265b67104e3f9e4fe695a08fcc"} Oct 29 23:16:57.655841 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:57.655781 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" podStartSLOduration=1.655767368 podStartE2EDuration="1.655767368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:16:57.652626754 +0000 UTC m=+2071.593174665" watchObservedRunningTime="2026-10-29 23:16:57.655767368 +0000 UTC m=+2071.596315277" Oct 29 23:16:57.666454 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:57.666424 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:16:58.481840 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:58.481803 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Oct 29 23:16:58.482403 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:58.482022 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" podUID="3b626717-62b8-428e-8371-9244e9f3e55f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Oct 29 23:16:58.673945 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:58.673910 4821 patch_prober.go:29] interesting pod/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 29 23:16:58.674221 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:58.674170 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" podUID="4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 29 23:16:59.283830 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.277451 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd"] Oct 29 23:16:59.285482 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.285448 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" Oct 29 23:16:59.290937 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.290478 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd"] Oct 29 23:16:59.348589 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.348543 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ea8e77\"/\"kube-root-ca.crt\"" Oct 29 23:16:59.348947 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.348921 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ea8e77\"/\"openshift-service-ca.crt\"" Oct 29 23:16:59.349265 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.349249 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ea8e77\"/\"default-dockercfg-lmdhf\"" Oct 29 23:16:59.351247 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.351222 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ea8e77\"/\"noten-model-ea8e77-kserve-self-signed-certs\"" Oct 29 23:16:59.400028 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.399988 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhcrt\" (UniqueName: \"kubernetes.io/projected/9b698c26-8020-47cb-a871-c558dabf4065-kube-api-access-vhcrt\") pod \"noten-model-ea8e77-kserve-788b78985c-8mbtd\" (UID: \"9b698c26-8020-47cb-a871-c558dabf4065\") " pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" Oct 29 23:16:59.400363 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.400340 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-home\") pod \"noten-model-ea8e77-kserve-788b78985c-8mbtd\" (UID: \"9b698c26-8020-47cb-a871-c558dabf4065\") " pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" Oct 29 23:16:59.400590 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.400571 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-dshm\") pod \"noten-model-ea8e77-kserve-788b78985c-8mbtd\" (UID: \"9b698c26-8020-47cb-a871-c558dabf4065\") " pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" Oct 29 23:16:59.400803 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.400785 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-tmp-dir\") pod \"noten-model-ea8e77-kserve-788b78985c-8mbtd\" (UID: \"9b698c26-8020-47cb-a871-c558dabf4065\") " pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" Oct 29 23:16:59.400937 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.400920 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-model-cache\") pod \"noten-model-ea8e77-kserve-788b78985c-8mbtd\" (UID: \"9b698c26-8020-47cb-a871-c558dabf4065\") " pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" Oct 29 23:16:59.401090 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.401077 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b698c26-8020-47cb-a871-c558dabf4065-tls-certs\") pod \"noten-model-ea8e77-kserve-788b78985c-8mbtd\" (UID: \"9b698c26-8020-47cb-a871-c558dabf4065\") " pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" Oct 29 23:16:59.477628 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.477593 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Oct 29 23:16:59.479768 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.478902 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" podUID="3b626717-62b8-428e-8371-9244e9f3e55f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Oct 29 23:16:59.505788 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.505181 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhcrt\" (UniqueName: \"kubernetes.io/projected/9b698c26-8020-47cb-a871-c558dabf4065-kube-api-access-vhcrt\") pod \"noten-model-ea8e77-kserve-788b78985c-8mbtd\" (UID: \"9b698c26-8020-47cb-a871-c558dabf4065\") " pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" Oct 29 23:16:59.506966 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.506390 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-home\") pod \"noten-model-ea8e77-kserve-788b78985c-8mbtd\" (UID: \"9b698c26-8020-47cb-a871-c558dabf4065\") " pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" Oct 29 23:16:59.507318 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.506938 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-home\") pod \"noten-model-ea8e77-kserve-788b78985c-8mbtd\" (UID: \"9b698c26-8020-47cb-a871-c558dabf4065\") " pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" Oct 29 23:16:59.507683 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.507630 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-dshm\") pod \"noten-model-ea8e77-kserve-788b78985c-8mbtd\" (UID: \"9b698c26-8020-47cb-a871-c558dabf4065\") " pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" Oct 29 23:16:59.508505 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.508486 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-tmp-dir\") pod \"noten-model-ea8e77-kserve-788b78985c-8mbtd\" (UID: \"9b698c26-8020-47cb-a871-c558dabf4065\") " pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" Oct 29 23:16:59.508981 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.508962 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-model-cache\") pod \"noten-model-ea8e77-kserve-788b78985c-8mbtd\" (UID: \"9b698c26-8020-47cb-a871-c558dabf4065\") " pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" Oct 29 23:16:59.509531 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.509515 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b698c26-8020-47cb-a871-c558dabf4065-tls-certs\") pod \"noten-model-ea8e77-kserve-788b78985c-8mbtd\" (UID: \"9b698c26-8020-47cb-a871-c558dabf4065\") " pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" Oct 29 23:16:59.510257 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.508897 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-tmp-dir\") pod \"noten-model-ea8e77-kserve-788b78985c-8mbtd\" (UID: \"9b698c26-8020-47cb-a871-c558dabf4065\") " pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" Oct 29 23:16:59.510414 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.509418 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-model-cache\") pod \"noten-model-ea8e77-kserve-788b78985c-8mbtd\" (UID: \"9b698c26-8020-47cb-a871-c558dabf4065\") " pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" Oct 29 23:16:59.564281 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.535821 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b698c26-8020-47cb-a871-c558dabf4065-tls-certs\") pod \"noten-model-ea8e77-kserve-788b78985c-8mbtd\" (UID: \"9b698c26-8020-47cb-a871-c558dabf4065\") " pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" Oct 29 23:16:59.564281 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.540976 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhcrt\" (UniqueName: \"kubernetes.io/projected/9b698c26-8020-47cb-a871-c558dabf4065-kube-api-access-vhcrt\") pod \"noten-model-ea8e77-kserve-788b78985c-8mbtd\" (UID: \"9b698c26-8020-47cb-a871-c558dabf4065\") " pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" Oct 29 23:16:59.564281 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.542045 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-dshm\") pod \"noten-model-ea8e77-kserve-788b78985c-8mbtd\" (UID: \"9b698c26-8020-47cb-a871-c558dabf4065\") " pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" Oct 29 23:16:59.622344 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.619813 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" Oct 29 23:16:59.666764 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.666302 4821 patch_prober.go:29] interesting pod/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 29 23:16:59.666764 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:16:59.666367 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" podUID="4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 29 23:17:00.279638 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:17:00.279600 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b698c26_8020_47cb_a871_c558dabf4065.slice/crio-5fc94ff14fa6745c379f68b5adf386fb780b6fceecba87d894a1640af9e1a830 WatchSource:0}: Error finding container 5fc94ff14fa6745c379f68b5adf386fb780b6fceecba87d894a1640af9e1a830: Status 404 returned error can't find the container with id 5fc94ff14fa6745c379f68b5adf386fb780b6fceecba87d894a1640af9e1a830 Oct 29 23:17:00.279912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:00.279879 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd"] Oct 29 23:17:00.483226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:00.482864 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Oct 29 23:17:00.483226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:00.482928 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" podUID="3b626717-62b8-428e-8371-9244e9f3e55f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Oct 29 23:17:00.666501 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:00.666467 4821 patch_prober.go:29] interesting pod/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 29 23:17:00.666915 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:00.666525 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" podUID="4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 29 23:17:00.755886 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:00.755845 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" event={"ID":"9b698c26-8020-47cb-a871-c558dabf4065","Type":"ContainerStarted","Data":"396465717e01905e575f051bc122b259cec7a7af3d36a9f373deceb88b9eb495"} Oct 29 23:17:00.756073 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:00.755904 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" event={"ID":"9b698c26-8020-47cb-a871-c558dabf4065","Type":"ContainerStarted","Data":"5fc94ff14fa6745c379f68b5adf386fb780b6fceecba87d894a1640af9e1a830"} Oct 29 23:17:00.805240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:00.805154 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" podStartSLOduration=1.805134551 podStartE2EDuration="1.805134551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:00.801972271 +0000 UTC m=+2074.742520182" watchObservedRunningTime="2026-10-29 23:17:00.805134551 +0000 UTC m=+2074.745682461" Oct 29 23:17:01.481590 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:01.481557 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:17:01.482074 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:01.482056 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:17:01.482928 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:01.482909 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:17:01.646494 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:01.646461 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" Oct 29 23:17:01.666302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:01.666269 4821 patch_prober.go:29] interesting pod/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 29 23:17:01.666506 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:01.666332 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" podUID="4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 29 23:17:01.875112 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:01.875071 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4"] Oct 29 23:17:01.879906 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:01.879886 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" Oct 29 23:17:01.902068 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:01.902005 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-faa4fc-serving-cert\"" Oct 29 23:17:01.907622 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:01.907594 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4"] Oct 29 23:17:01.973829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:01.973793 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-maas-api-tls\") pod \"maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4\" (UID: \"ce5e06a7-dc83-4262-a6a4-ead5285a1dd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" Oct 29 23:17:01.974001 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:01.973845 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-metrics-certs\") pod \"maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4\" (UID: \"ce5e06a7-dc83-4262-a6a4-ead5285a1dd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" Oct 29 23:17:01.974077 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:01.974056 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfr4w\" (UniqueName: \"kubernetes.io/projected/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-kube-api-access-sfr4w\") pod \"maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4\" (UID: \"ce5e06a7-dc83-4262-a6a4-ead5285a1dd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" Oct 29 23:17:01.974278 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:01.974258 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4\" (UID: \"ce5e06a7-dc83-4262-a6a4-ead5285a1dd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" Oct 29 23:17:02.075735 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:02.075691 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4\" (UID: \"ce5e06a7-dc83-4262-a6a4-ead5285a1dd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" Oct 29 23:17:02.075924 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:02.075810 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-maas-api-tls\") pod \"maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4\" (UID: \"ce5e06a7-dc83-4262-a6a4-ead5285a1dd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" Oct 29 23:17:02.075924 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:02.075840 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-metrics-certs\") pod \"maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4\" (UID: \"ce5e06a7-dc83-4262-a6a4-ead5285a1dd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" Oct 29 23:17:02.076005 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:02.075984 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfr4w\" (UniqueName: \"kubernetes.io/projected/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-kube-api-access-sfr4w\") pod \"maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4\" (UID: \"ce5e06a7-dc83-4262-a6a4-ead5285a1dd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" Oct 29 23:17:02.076506 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:02.076483 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4\" (UID: \"ce5e06a7-dc83-4262-a6a4-ead5285a1dd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" Oct 29 23:17:02.078545 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:02.078524 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-metrics-certs\") pod \"maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4\" (UID: \"ce5e06a7-dc83-4262-a6a4-ead5285a1dd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" Oct 29 23:17:02.078711 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:02.078692 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-maas-api-tls\") pod \"maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4\" (UID: \"ce5e06a7-dc83-4262-a6a4-ead5285a1dd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" Oct 29 23:17:02.101418 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:02.101389 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfr4w\" (UniqueName: \"kubernetes.io/projected/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-kube-api-access-sfr4w\") pod \"maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4\" (UID: \"ce5e06a7-dc83-4262-a6a4-ead5285a1dd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" Oct 29 23:17:02.207296 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:02.202552 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" Oct 29 23:17:02.566016 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:17:02.565436 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce5e06a7_dc83_4262_a6a4_ead5285a1dd3.slice/crio-8b0a38d30f6631516a0b9381b0e5c29fa4ed3319c4b6067cb16d6dab9d5403aa WatchSource:0}: Error finding container 8b0a38d30f6631516a0b9381b0e5c29fa4ed3319c4b6067cb16d6dab9d5403aa: Status 404 returned error can't find the container with id 8b0a38d30f6631516a0b9381b0e5c29fa4ed3319c4b6067cb16d6dab9d5403aa Oct 29 23:17:02.566016 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:02.565820 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4"] Oct 29 23:17:02.666599 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:02.666565 4821 patch_prober.go:29] interesting pod/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 29 23:17:02.666771 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:02.666623 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" podUID="4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 29 23:17:02.775888 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:02.775835 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" event={"ID":"ce5e06a7-dc83-4262-a6a4-ead5285a1dd3","Type":"ContainerStarted","Data":"8b0a38d30f6631516a0b9381b0e5c29fa4ed3319c4b6067cb16d6dab9d5403aa"} Oct 29 23:17:03.667123 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:03.667080 4821 patch_prober.go:29] interesting pod/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 29 23:17:03.667661 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:03.667152 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" podUID="4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 29 23:17:03.790954 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:03.790913 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" event={"ID":"ce5e06a7-dc83-4262-a6a4-ead5285a1dd3","Type":"ContainerStarted","Data":"f99823ad6c628c1498aa1709251b5efb71b710b9a2de83863ead13fe6f059c84"} Oct 29 23:17:03.791150 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:03.791013 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" Oct 29 23:17:03.838908 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:03.838843 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" podStartSLOduration=2.504548887 podStartE2EDuration="2.838826124s" totalImagesPullingTime="334.277237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:03.835939978 +0000 UTC m=+2077.776487879" watchObservedRunningTime="2026-10-29 23:17:03.838826124 +0000 UTC m=+2077.779374033" Oct 29 23:17:04.672525 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:04.672492 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:17:04.673054 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:04.673029 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:17:04.673859 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:04.673836 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:17:04.871597 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:04.871562 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd"] Oct 29 23:17:05.314233 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:05.305870 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp"] Oct 29 23:17:05.338041 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:05.337997 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" podUID="b144b55c-1c80-469b-ba68-671fdc2e879f" containerName="istio-proxy" containerID="cri-o://3bf5211ecf3874f40e0fd0b7b56a7633f072a75a8cc6d15a5eda2ce46798f0d1" gracePeriod=30 Oct 29 23:17:09.807431 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:09.807401 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" Oct 29 23:17:10.671424 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.671356 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:17:10.757794 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.757760 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-envoy\") pod \"b144b55c-1c80-469b-ba68-671fdc2e879f\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " Oct 29 23:17:10.758006 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.757811 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-token\" (UniqueName: \"kubernetes.io/projected/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-token\") pod \"b144b55c-1c80-469b-ba68-671fdc2e879f\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " Oct 29 23:17:10.758006 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.757829 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-data\") pod \"b144b55c-1c80-469b-ba68-671fdc2e879f\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " Oct 29 23:17:10.758006 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.757914 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-podinfo\") pod \"b144b55c-1c80-469b-ba68-671fdc2e879f\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " Oct 29 23:17:10.758006 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.757942 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b144b55c-1c80-469b-ba68-671fdc2e879f-kube-api-access-xqhpf\" (UniqueName: \"kubernetes.io/projected/b144b55c-1c80-469b-ba68-671fdc2e879f-kube-api-access-xqhpf\") pod \"b144b55c-1c80-469b-ba68-671fdc2e879f\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " Oct 29 23:17:10.758006 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.757990 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-workload-socket\") pod \"b144b55c-1c80-469b-ba68-671fdc2e879f\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " Oct 29 23:17:10.758382 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.758013 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-workload-certs\") pod \"b144b55c-1c80-469b-ba68-671fdc2e879f\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " Oct 29 23:17:10.758382 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.758087 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-credential-socket\") pod \"b144b55c-1c80-469b-ba68-671fdc2e879f\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " Oct 29 23:17:10.758382 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.758356 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b144b55c-1c80-469b-ba68-671fdc2e879f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b144b55c-1c80-469b-ba68-671fdc2e879f-istiod-ca-cert\") pod \"b144b55c-1c80-469b-ba68-671fdc2e879f\" (UID: \"b144b55c-1c80-469b-ba68-671fdc2e879f\") " Oct 29 23:17:10.758968 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.758938 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-workload-socket" pod "b144b55c-1c80-469b-ba68-671fdc2e879f" (UID: "b144b55c-1c80-469b-ba68-671fdc2e879f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:17:10.759344 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.759305 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b144b55c-1c80-469b-ba68-671fdc2e879f-istiod-ca-cert" pod "b144b55c-1c80-469b-ba68-671fdc2e879f" (UID: "b144b55c-1c80-469b-ba68-671fdc2e879f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:17:10.759545 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.759520 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:17:10.759639 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.759544 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b144b55c-1c80-469b-ba68-671fdc2e879f-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:17:10.759836 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.759811 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-workload-certs" pod "b144b55c-1c80-469b-ba68-671fdc2e879f" (UID: "b144b55c-1c80-469b-ba68-671fdc2e879f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:17:10.760053 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.760025 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-data" pod "b144b55c-1c80-469b-ba68-671fdc2e879f" (UID: "b144b55c-1c80-469b-ba68-671fdc2e879f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:17:10.760053 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.760040 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-credential-socket" pod "b144b55c-1c80-469b-ba68-671fdc2e879f" (UID: "b144b55c-1c80-469b-ba68-671fdc2e879f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:17:10.760803 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.760776 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-envoy" pod "b144b55c-1c80-469b-ba68-671fdc2e879f" (UID: "b144b55c-1c80-469b-ba68-671fdc2e879f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:17:10.761747 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.761722 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-podinfo" pod "b144b55c-1c80-469b-ba68-671fdc2e879f" (UID: "b144b55c-1c80-469b-ba68-671fdc2e879f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:17:10.767033 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.767007 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b144b55c-1c80-469b-ba68-671fdc2e879f-kube-api-access-xqhpf" pod "b144b55c-1c80-469b-ba68-671fdc2e879f" (UID: "b144b55c-1c80-469b-ba68-671fdc2e879f"). InnerVolumeSpecName "kube-api-access-xqhpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:17:10.769673 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.769642 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-token" pod "b144b55c-1c80-469b-ba68-671fdc2e879f" (UID: "b144b55c-1c80-469b-ba68-671fdc2e879f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:17:10.853410 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.853372 4821 generic.go:356] "Generic (PLEG): container finished" podID="b144b55c-1c80-469b-ba68-671fdc2e879f" containerID="3bf5211ecf3874f40e0fd0b7b56a7633f072a75a8cc6d15a5eda2ce46798f0d1" exitCode=0 Oct 29 23:17:10.853827 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.853425 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" Oct 29 23:17:10.853827 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.853453 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" event={"ID":"b144b55c-1c80-469b-ba68-671fdc2e879f","Type":"ContainerDied","Data":"3bf5211ecf3874f40e0fd0b7b56a7633f072a75a8cc6d15a5eda2ce46798f0d1"} Oct 29 23:17:10.853827 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.853505 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp" event={"ID":"b144b55c-1c80-469b-ba68-671fdc2e879f","Type":"ContainerDied","Data":"4e80bc4e31db75a0105e4de6f86f058f4d542ea127848e32a10a19246aa8f501"} Oct 29 23:17:10.853827 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.853526 4821 scope.go:122] "RemoveContainer" containerID="3bf5211ecf3874f40e0fd0b7b56a7633f072a75a8cc6d15a5eda2ce46798f0d1" Oct 29 23:17:10.860679 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.860658 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:17:10.860811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.860681 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqhpf\" (UniqueName: \"kubernetes.io/projected/b144b55c-1c80-469b-ba68-671fdc2e879f-kube-api-access-xqhpf\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:17:10.860811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.860696 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:17:10.860811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.860710 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:17:10.860811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.860722 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:17:10.860811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.860735 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:17:10.860811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.860746 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b144b55c-1c80-469b-ba68-671fdc2e879f-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:17:10.865826 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.865810 4821 scope.go:122] "RemoveContainer" containerID="3bf5211ecf3874f40e0fd0b7b56a7633f072a75a8cc6d15a5eda2ce46798f0d1" Oct 29 23:17:10.866114 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:17:10.866096 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf5211ecf3874f40e0fd0b7b56a7633f072a75a8cc6d15a5eda2ce46798f0d1\": container with ID starting with 3bf5211ecf3874f40e0fd0b7b56a7633f072a75a8cc6d15a5eda2ce46798f0d1 not found: ID does not exist" containerID="3bf5211ecf3874f40e0fd0b7b56a7633f072a75a8cc6d15a5eda2ce46798f0d1" Oct 29 23:17:10.866179 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.866126 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf5211ecf3874f40e0fd0b7b56a7633f072a75a8cc6d15a5eda2ce46798f0d1"} err="failed to get container status \"3bf5211ecf3874f40e0fd0b7b56a7633f072a75a8cc6d15a5eda2ce46798f0d1\": rpc error: code = NotFound desc = could not find container \"3bf5211ecf3874f40e0fd0b7b56a7633f072a75a8cc6d15a5eda2ce46798f0d1\": container with ID starting with 3bf5211ecf3874f40e0fd0b7b56a7633f072a75a8cc6d15a5eda2ce46798f0d1 not found: ID does not exist" Oct 29 23:17:10.916874 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.916839 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp"] Oct 29 23:17:10.924377 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:10.924301 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-ea8e77-openshift-default-5b7c9bd495-6fmbp"] Oct 29 23:17:11.469845 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.469808 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5"] Oct 29 23:17:11.471172 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.471131 4821 cpu_manager.go:414] "containerMap: removing container" podUID="b144b55c-1c80-469b-ba68-671fdc2e879f" containerName="istio-proxy" Oct 29 23:17:11.471172 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.471166 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b144b55c-1c80-469b-ba68-671fdc2e879f" containerName="istio-proxy" Oct 29 23:17:11.471768 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.471690 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="b144b55c-1c80-469b-ba68-671fdc2e879f" containerName="istio-proxy" Oct 29 23:17:11.477523 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.477500 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.497845 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.496850 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5"] Oct 29 23:17:11.503066 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.503041 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b2b71e-openshift-default-dockercfg-8n8b9\"" Oct 29 23:17:11.571841 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.571795 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-workload-certs\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.572026 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.571969 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-credential-socket\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.572107 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.572022 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istiod-ca-cert\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.572107 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.572047 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-envoy\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.572289 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.572129 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-podinfo\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.572353 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.572309 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-workload-socket\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.572353 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.572331 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-data\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.572477 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.572370 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-token\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.572477 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.572445 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct7nx\" (UniqueName: \"kubernetes.io/projected/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-kube-api-access-ct7nx\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.688375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.686622 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-workload-socket\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.688642 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.687080 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-workload-socket\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.688819 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.688799 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-data\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.688988 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.688967 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-token\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.689127 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.689111 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct7nx\" (UniqueName: \"kubernetes.io/projected/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-kube-api-access-ct7nx\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.689587 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.689565 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-data\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.713173 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.713139 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-workload-certs\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.713676 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.713645 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-workload-certs\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.713873 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.713851 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-credential-socket\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.714064 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.714036 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istiod-ca-cert\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.735286 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.714242 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-envoy\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.735652 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.714694 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-credential-socket\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.735784 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.714896 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istiod-ca-cert\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.735869 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.735644 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-podinfo\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.758227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.736985 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-token\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.758227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.739834 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct7nx\" (UniqueName: \"kubernetes.io/projected/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-kube-api-access-ct7nx\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.758227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.739946 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-podinfo\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.766222 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.763518 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-envoy\") pod \"e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:11.797357 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:11.796538 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:12.175146 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:12.175112 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b144b55c-1c80-469b-ba68-671fdc2e879f" path="/var/lib/kubelet/pods/b144b55c-1c80-469b-ba68-671fdc2e879f/volumes" Oct 29 23:17:12.312148 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:12.312116 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5"] Oct 29 23:17:12.312864 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:17:12.312833 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf2ae83e_0943_4642_aa5f_2378b18bfe9d.slice/crio-d6eb9b4554bdace69af8ea372fce54b55f494af01c72c6f92c20e973bddea0f9 WatchSource:0}: Error finding container d6eb9b4554bdace69af8ea372fce54b55f494af01c72c6f92c20e973bddea0f9: Status 404 returned error can't find the container with id d6eb9b4554bdace69af8ea372fce54b55f494af01c72c6f92c20e973bddea0f9 Oct 29 23:17:12.316216 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:12.316162 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:17:12.316315 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:12.316270 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:17:12.316315 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:12.316304 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:17:12.920986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:12.920950 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" event={"ID":"cf2ae83e-0943-4642-aa5f-2378b18bfe9d","Type":"ContainerStarted","Data":"f14eeccb9ce53195c36db0341645d92bdfbb84d24077d5e4f8723b67cc84a96d"} Oct 29 23:17:12.921329 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:12.921015 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" event={"ID":"cf2ae83e-0943-4642-aa5f-2378b18bfe9d","Type":"ContainerStarted","Data":"d6eb9b4554bdace69af8ea372fce54b55f494af01c72c6f92c20e973bddea0f9"} Oct 29 23:17:12.985944 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:12.985889 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" podStartSLOduration=1.985870903 podStartE2EDuration="1.985870903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:12.985069092 +0000 UTC m=+2086.925617002" watchObservedRunningTime="2026-10-29 23:17:12.985870903 +0000 UTC m=+2086.926418820" Oct 29 23:17:13.797602 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:13.797568 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:13.805043 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:13.805014 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:13.930916 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:13.930883 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:13.931909 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:13.931882 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:17:17.019734 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:17.019697 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk"] Oct 29 23:17:17.025688 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:17.025665 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk" Oct 29 23:17:17.053430 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:17.053400 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk"] Oct 29 23:17:17.096050 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:17.087578 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b2b71e\"" Oct 29 23:17:17.120224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:17.098610 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b2b71e-dockercfg-zqndz\"" Oct 29 23:17:17.328856 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:17.328758 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52400e6-b2db-4dca-b23d-ad9f9a81bf44-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk\" (UID: \"e52400e6-b2db-4dca-b23d-ad9f9a81bf44\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk" Oct 29 23:17:17.329043 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:17.329020 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmkfv\" (UniqueName: \"kubernetes.io/projected/e52400e6-b2db-4dca-b23d-ad9f9a81bf44-kube-api-access-vmkfv\") pod \"payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk\" (UID: \"e52400e6-b2db-4dca-b23d-ad9f9a81bf44\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk" Oct 29 23:17:17.431759 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:17.431720 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52400e6-b2db-4dca-b23d-ad9f9a81bf44-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk\" (UID: \"e52400e6-b2db-4dca-b23d-ad9f9a81bf44\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk" Oct 29 23:17:17.431972 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:17.431795 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmkfv\" (UniqueName: \"kubernetes.io/projected/e52400e6-b2db-4dca-b23d-ad9f9a81bf44-kube-api-access-vmkfv\") pod \"payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk\" (UID: \"e52400e6-b2db-4dca-b23d-ad9f9a81bf44\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk" Oct 29 23:17:17.432763 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:17.432734 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52400e6-b2db-4dca-b23d-ad9f9a81bf44-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk\" (UID: \"e52400e6-b2db-4dca-b23d-ad9f9a81bf44\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk" Oct 29 23:17:17.469787 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:17.469752 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmkfv\" (UniqueName: \"kubernetes.io/projected/e52400e6-b2db-4dca-b23d-ad9f9a81bf44-kube-api-access-vmkfv\") pod \"payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk\" (UID: \"e52400e6-b2db-4dca-b23d-ad9f9a81bf44\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk" Oct 29 23:17:17.647164 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:17.647128 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk" Oct 29 23:17:17.985253 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:17.978620 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk"] Oct 29 23:17:17.990467 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:17:17.990259 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode52400e6_b2db_4dca_b23d_ad9f9a81bf44.slice/crio-5843dd4f1ebb97d1c167134b7d8bc9f92b900803a114e003ac36d72e1baa5b61 WatchSource:0}: Error finding container 5843dd4f1ebb97d1c167134b7d8bc9f92b900803a114e003ac36d72e1baa5b61: Status 404 returned error can't find the container with id 5843dd4f1ebb97d1c167134b7d8bc9f92b900803a114e003ac36d72e1baa5b61 Oct 29 23:17:18.995485 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:18.995439 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk" event={"ID":"e52400e6-b2db-4dca-b23d-ad9f9a81bf44","Type":"ContainerStarted","Data":"0ed57a09dd98f3049981c1e8c26b799da8836aeb5e34be23f0d3dbaf98895f9d"} Oct 29 23:17:18.995960 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:18.995524 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk" event={"ID":"e52400e6-b2db-4dca-b23d-ad9f9a81bf44","Type":"ContainerStarted","Data":"5843dd4f1ebb97d1c167134b7d8bc9f92b900803a114e003ac36d72e1baa5b61"} Oct 29 23:17:18.995960 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:18.995556 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk" Oct 29 23:17:19.043113 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:19.043053 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk" podStartSLOduration=3.043038879 podStartE2EDuration="3.043038879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:19.041268467 +0000 UTC m=+2092.981816379" watchObservedRunningTime="2026-10-29 23:17:19.043038879 +0000 UTC m=+2092.983586789" Oct 29 23:17:19.922653 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:19.922601 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" podUID="9b698c26-8020-47cb-a871-c558dabf4065" containerName="main" containerID="cri-o://396465717e01905e575f051bc122b259cec7a7af3d36a9f373deceb88b9eb495" gracePeriod=45 Oct 29 23:17:20.234025 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:20.234000 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" Oct 29 23:17:20.286220 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:20.286171 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-tmp-dir\") pod \"9b698c26-8020-47cb-a871-c558dabf4065\" (UID: \"9b698c26-8020-47cb-a871-c558dabf4065\") " Oct 29 23:17:20.286404 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:20.286310 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b698c26-8020-47cb-a871-c558dabf4065-kube-api-access-vhcrt\" (UniqueName: \"kubernetes.io/projected/9b698c26-8020-47cb-a871-c558dabf4065-kube-api-access-vhcrt\") pod \"9b698c26-8020-47cb-a871-c558dabf4065\" (UID: \"9b698c26-8020-47cb-a871-c558dabf4065\") " Oct 29 23:17:20.286404 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:20.286355 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-home\" (UniqueName: \"kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-home\") pod \"9b698c26-8020-47cb-a871-c558dabf4065\" (UID: \"9b698c26-8020-47cb-a871-c558dabf4065\") " Oct 29 23:17:20.286521 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:20.286489 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-tmp-dir" pod "9b698c26-8020-47cb-a871-c558dabf4065" (UID: "9b698c26-8020-47cb-a871-c558dabf4065"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:17:20.286583 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:20.286537 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b698c26-8020-47cb-a871-c558dabf4065-tls-certs\" (UniqueName: \"kubernetes.io/secret/9b698c26-8020-47cb-a871-c558dabf4065-tls-certs\") pod \"9b698c26-8020-47cb-a871-c558dabf4065\" (UID: \"9b698c26-8020-47cb-a871-c558dabf4065\") " Oct 29 23:17:20.286672 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:20.286641 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-home" pod "9b698c26-8020-47cb-a871-c558dabf4065" (UID: "9b698c26-8020-47cb-a871-c558dabf4065"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:17:20.286733 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:20.286674 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-model-cache\") pod \"9b698c26-8020-47cb-a871-c558dabf4065\" (UID: \"9b698c26-8020-47cb-a871-c558dabf4065\") " Oct 29 23:17:20.286790 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:20.286742 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-dshm\") pod \"9b698c26-8020-47cb-a871-c558dabf4065\" (UID: \"9b698c26-8020-47cb-a871-c558dabf4065\") " Oct 29 23:17:20.286959 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:20.286935 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-model-cache" pod "9b698c26-8020-47cb-a871-c558dabf4065" (UID: "9b698c26-8020-47cb-a871-c558dabf4065"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:17:20.287605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:20.287583 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:17:20.287605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:20.287605 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:17:20.287764 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:20.287617 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:17:20.288938 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:20.288914 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b698c26-8020-47cb-a871-c558dabf4065-kube-api-access-vhcrt" pod "9b698c26-8020-47cb-a871-c558dabf4065" (UID: "9b698c26-8020-47cb-a871-c558dabf4065"). InnerVolumeSpecName "kube-api-access-vhcrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:17:20.289126 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:20.289108 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b698c26-8020-47cb-a871-c558dabf4065-tls-certs" pod "9b698c26-8020-47cb-a871-c558dabf4065" (UID: "9b698c26-8020-47cb-a871-c558dabf4065"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:17:20.289216 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:20.289174 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-dshm" pod "9b698c26-8020-47cb-a871-c558dabf4065" (UID: "9b698c26-8020-47cb-a871-c558dabf4065"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:17:20.389280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:20.389244 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhcrt\" (UniqueName: \"kubernetes.io/projected/9b698c26-8020-47cb-a871-c558dabf4065-kube-api-access-vhcrt\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:17:20.389280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:20.389274 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b698c26-8020-47cb-a871-c558dabf4065-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:17:20.389280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:20.389282 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b698c26-8020-47cb-a871-c558dabf4065-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:17:21.015144 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:21.015110 4821 generic.go:356] "Generic (PLEG): container finished" podID="9b698c26-8020-47cb-a871-c558dabf4065" containerID="396465717e01905e575f051bc122b259cec7a7af3d36a9f373deceb88b9eb495" exitCode=0 Oct 29 23:17:21.015337 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:21.015186 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" Oct 29 23:17:21.015337 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:21.015189 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" event={"ID":"9b698c26-8020-47cb-a871-c558dabf4065","Type":"ContainerDied","Data":"396465717e01905e575f051bc122b259cec7a7af3d36a9f373deceb88b9eb495"} Oct 29 23:17:21.015337 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:21.015266 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd" event={"ID":"9b698c26-8020-47cb-a871-c558dabf4065","Type":"ContainerDied","Data":"5fc94ff14fa6745c379f68b5adf386fb780b6fceecba87d894a1640af9e1a830"} Oct 29 23:17:21.015337 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:21.015283 4821 scope.go:122] "RemoveContainer" containerID="396465717e01905e575f051bc122b259cec7a7af3d36a9f373deceb88b9eb495" Oct 29 23:17:21.027726 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:21.027705 4821 scope.go:122] "RemoveContainer" containerID="396465717e01905e575f051bc122b259cec7a7af3d36a9f373deceb88b9eb495" Oct 29 23:17:21.028016 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:17:21.027990 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"396465717e01905e575f051bc122b259cec7a7af3d36a9f373deceb88b9eb495\": container with ID starting with 396465717e01905e575f051bc122b259cec7a7af3d36a9f373deceb88b9eb495 not found: ID does not exist" containerID="396465717e01905e575f051bc122b259cec7a7af3d36a9f373deceb88b9eb495" Oct 29 23:17:21.028075 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:21.028030 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396465717e01905e575f051bc122b259cec7a7af3d36a9f373deceb88b9eb495"} err="failed to get container status \"396465717e01905e575f051bc122b259cec7a7af3d36a9f373deceb88b9eb495\": rpc error: code = NotFound desc = could not find container \"396465717e01905e575f051bc122b259cec7a7af3d36a9f373deceb88b9eb495\": container with ID starting with 396465717e01905e575f051bc122b259cec7a7af3d36a9f373deceb88b9eb495 not found: ID does not exist" Oct 29 23:17:21.074854 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:21.074823 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd"] Oct 29 23:17:21.087608 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:21.087578 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-ea8e77/noten-model-ea8e77-kserve-788b78985c-8mbtd"] Oct 29 23:17:22.172339 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:22.172306 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b698c26-8020-47cb-a871-c558dabf4065" path="/var/lib/kubelet/pods/9b698c26-8020-47cb-a871-c558dabf4065/volumes" Oct 29 23:17:23.887978 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:17:23.887938 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:17:30.008705 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:30.008674 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk" Oct 29 23:17:30.905306 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:30.905263 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw"] Oct 29 23:17:30.906081 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:30.906063 4821 cpu_manager.go:414] "containerMap: removing container" podUID="9b698c26-8020-47cb-a871-c558dabf4065" containerName="main" Oct 29 23:17:30.906081 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:30.906081 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b698c26-8020-47cb-a871-c558dabf4065" containerName="main" Oct 29 23:17:30.906299 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:30.906229 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b698c26-8020-47cb-a871-c558dabf4065" containerName="main" Oct 29 23:17:30.911845 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:30.911825 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw" Oct 29 23:17:30.938759 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:30.938729 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f4ce70-serving-cert\"" Oct 29 23:17:30.946398 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:30.946366 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw"] Oct 29 23:17:30.971636 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:30.971602 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41006044-c141-447c-8707-cf9d8d541c6b-metrics-certs\") pod \"maas-api-e2e-inf-b-f4ce70-66b578946-wxghw\" (UID: \"41006044-c141-447c-8707-cf9d8d541c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw" Oct 29 23:17:30.971807 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:30.971777 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41006044-c141-447c-8707-cf9d8d541c6b-maas-api-tls\") pod \"maas-api-e2e-inf-b-f4ce70-66b578946-wxghw\" (UID: \"41006044-c141-447c-8707-cf9d8d541c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw" Oct 29 23:17:30.971857 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:30.971821 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-749zx\" (UniqueName: \"kubernetes.io/projected/41006044-c141-447c-8707-cf9d8d541c6b-kube-api-access-749zx\") pod \"maas-api-e2e-inf-b-f4ce70-66b578946-wxghw\" (UID: \"41006044-c141-447c-8707-cf9d8d541c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw" Oct 29 23:17:30.972044 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:30.972022 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41006044-c141-447c-8707-cf9d8d541c6b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f4ce70-66b578946-wxghw\" (UID: \"41006044-c141-447c-8707-cf9d8d541c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw" Oct 29 23:17:31.078152 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:31.073324 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41006044-c141-447c-8707-cf9d8d541c6b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f4ce70-66b578946-wxghw\" (UID: \"41006044-c141-447c-8707-cf9d8d541c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw" Oct 29 23:17:31.078152 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:31.073548 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41006044-c141-447c-8707-cf9d8d541c6b-metrics-certs\") pod \"maas-api-e2e-inf-b-f4ce70-66b578946-wxghw\" (UID: \"41006044-c141-447c-8707-cf9d8d541c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw" Oct 29 23:17:31.078152 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:31.073677 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41006044-c141-447c-8707-cf9d8d541c6b-maas-api-tls\") pod \"maas-api-e2e-inf-b-f4ce70-66b578946-wxghw\" (UID: \"41006044-c141-447c-8707-cf9d8d541c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw" Oct 29 23:17:31.078152 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:31.073728 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-749zx\" (UniqueName: \"kubernetes.io/projected/41006044-c141-447c-8707-cf9d8d541c6b-kube-api-access-749zx\") pod \"maas-api-e2e-inf-b-f4ce70-66b578946-wxghw\" (UID: \"41006044-c141-447c-8707-cf9d8d541c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw" Oct 29 23:17:31.078152 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:31.074947 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41006044-c141-447c-8707-cf9d8d541c6b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f4ce70-66b578946-wxghw\" (UID: \"41006044-c141-447c-8707-cf9d8d541c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw" Oct 29 23:17:31.101796 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:31.097599 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41006044-c141-447c-8707-cf9d8d541c6b-maas-api-tls\") pod \"maas-api-e2e-inf-b-f4ce70-66b578946-wxghw\" (UID: \"41006044-c141-447c-8707-cf9d8d541c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw" Oct 29 23:17:31.118227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:31.115889 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41006044-c141-447c-8707-cf9d8d541c6b-metrics-certs\") pod \"maas-api-e2e-inf-b-f4ce70-66b578946-wxghw\" (UID: \"41006044-c141-447c-8707-cf9d8d541c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw" Oct 29 23:17:31.127228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:31.127039 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-749zx\" (UniqueName: \"kubernetes.io/projected/41006044-c141-447c-8707-cf9d8d541c6b-kube-api-access-749zx\") pod \"maas-api-e2e-inf-b-f4ce70-66b578946-wxghw\" (UID: \"41006044-c141-447c-8707-cf9d8d541c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw" Oct 29 23:17:31.227362 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:31.227051 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw" Oct 29 23:17:31.563592 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:31.563506 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw"] Oct 29 23:17:31.567902 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:17:31.567873 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41006044_c141_447c_8707_cf9d8d541c6b.slice/crio-7c8926c02aba1eea6d2803ce9039e4677bb9adbbd9373faa8992c310a4eed0a4 WatchSource:0}: Error finding container 7c8926c02aba1eea6d2803ce9039e4677bb9adbbd9373faa8992c310a4eed0a4: Status 404 returned error can't find the container with id 7c8926c02aba1eea6d2803ce9039e4677bb9adbbd9373faa8992c310a4eed0a4 Oct 29 23:17:32.179481 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:32.179423 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw" event={"ID":"41006044-c141-447c-8707-cf9d8d541c6b","Type":"ContainerStarted","Data":"7c8926c02aba1eea6d2803ce9039e4677bb9adbbd9373faa8992c310a4eed0a4"} Oct 29 23:17:33.181943 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:33.181908 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw" event={"ID":"41006044-c141-447c-8707-cf9d8d541c6b","Type":"ContainerStarted","Data":"40bd4fccbcdbfd3728b0c4882237fbf4f6404cdb9bacffc319e039de79048fa1"} Oct 29 23:17:33.182472 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:33.182044 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw" Oct 29 23:17:33.228959 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:33.228908 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw" podStartSLOduration=2.836368194 podStartE2EDuration="3.228894777s" totalImagesPullingTime="392.526583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:33.226422248 +0000 UTC m=+2107.166970160" watchObservedRunningTime="2026-10-29 23:17:33.228894777 +0000 UTC m=+2107.169442686" Oct 29 23:17:39.206488 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:39.206387 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw" Oct 29 23:17:41.117456 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:41.117419 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7599-ce7135"] Oct 29 23:17:41.171409 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:41.171374 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7599-ce7135"] Oct 29 23:17:41.172721 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:41.172699 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-ce7135" Oct 29 23:17:41.206294 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:41.206260 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fvr2l\"" Oct 29 23:17:41.287770 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:41.287722 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zr2j\" (UniqueName: \"kubernetes.io/projected/4c9b5f23-a4b2-4d19-83c3-43e29f69071a-kube-api-access-6zr2j\") pod \"test-curl-7599-ce7135\" (UID: \"4c9b5f23-a4b2-4d19-83c3-43e29f69071a\") " pod="opendatahub/test-curl-7599-ce7135" Oct 29 23:17:41.391694 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:41.391132 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zr2j\" (UniqueName: \"kubernetes.io/projected/4c9b5f23-a4b2-4d19-83c3-43e29f69071a-kube-api-access-6zr2j\") pod \"test-curl-7599-ce7135\" (UID: \"4c9b5f23-a4b2-4d19-83c3-43e29f69071a\") " pod="opendatahub/test-curl-7599-ce7135" Oct 29 23:17:41.451734 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:41.451703 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zr2j\" (UniqueName: \"kubernetes.io/projected/4c9b5f23-a4b2-4d19-83c3-43e29f69071a-kube-api-access-6zr2j\") pod \"test-curl-7599-ce7135\" (UID: \"4c9b5f23-a4b2-4d19-83c3-43e29f69071a\") " pod="opendatahub/test-curl-7599-ce7135" Oct 29 23:17:41.508773 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:41.508737 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-ce7135" Oct 29 23:17:41.849340 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:17:41.849266 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c9b5f23_a4b2_4d19_83c3_43e29f69071a.slice/crio-aba4d4f6e844678f3aab06af0f45acff2548372cc239e65f5c9d1fe5a5d2d062 WatchSource:0}: Error finding container aba4d4f6e844678f3aab06af0f45acff2548372cc239e65f5c9d1fe5a5d2d062: Status 404 returned error can't find the container with id aba4d4f6e844678f3aab06af0f45acff2548372cc239e65f5c9d1fe5a5d2d062 Oct 29 23:17:41.853250 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:41.853175 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7599-ce7135"] Oct 29 23:17:42.550268 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:42.550234 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7599-ce7135" event={"ID":"4c9b5f23-a4b2-4d19-83c3-43e29f69071a","Type":"ContainerStarted","Data":"60a27500d456d90ea6ca235c8b4db6625b68cf4c8ce5f232da07d3aaf97c5878"} Oct 29 23:17:42.550673 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:42.550284 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7599-ce7135" event={"ID":"4c9b5f23-a4b2-4d19-83c3-43e29f69071a","Type":"ContainerStarted","Data":"aba4d4f6e844678f3aab06af0f45acff2548372cc239e65f5c9d1fe5a5d2d062"} Oct 29 23:17:42.602940 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:42.602884 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7599-ce7135" podStartSLOduration=1.602865731 podStartE2EDuration="1.602865731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:42.600255471 +0000 UTC m=+2116.540803382" watchObservedRunningTime="2026-10-29 23:17:42.602865731 +0000 UTC m=+2116.543413640" Oct 29 23:17:43.552010 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:43.551920 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7599-ce7135"] Oct 29 23:17:43.561292 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:43.559981 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7599-ce7135"] Oct 29 23:17:43.598775 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:43.598736 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f"] Oct 29 23:17:43.599936 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:43.599912 4821 cpu_manager.go:414] "containerMap: removing container" podUID="4c9b5f23-a4b2-4d19-83c3-43e29f69071a" containerName="test-curl-7599-ce7135" Oct 29 23:17:43.600017 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:43.599938 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c9b5f23-a4b2-4d19-83c3-43e29f69071a" containerName="test-curl-7599-ce7135" Oct 29 23:17:43.600173 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:43.600158 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c9b5f23-a4b2-4d19-83c3-43e29f69071a" containerName="test-curl-7599-ce7135" Oct 29 23:17:43.604962 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:43.604939 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:43.671818 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:43.671760 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a9ae14-openshift-default-dockercfg-fwsn6\"" Oct 29 23:17:43.673125 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:43.673089 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f"] Oct 29 23:17:43.787238 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:43.784357 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7599-15b816"] Oct 29 23:17:43.977394 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:43.952149 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-token\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:43.977394 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:43.952372 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-data\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:43.977394 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:43.952406 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-workload-certs\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:43.977394 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:43.952478 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istiod-ca-cert\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:43.977394 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:43.952520 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-envoy\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:43.977394 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:43.952575 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-credential-socket\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:43.977394 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:43.952615 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-podinfo\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:43.977394 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:43.952668 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-workload-socket\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:43.977394 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:43.952753 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fr6f\" (UniqueName: \"kubernetes.io/projected/9a986cc4-88d7-432a-8cc8-f2d5371373b1-kube-api-access-7fr6f\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:43.996390 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:43.996339 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7599-15b816"] Oct 29 23:17:43.996730 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:43.996712 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-15b816" Oct 29 23:17:44.091227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.089128 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fr6f\" (UniqueName: \"kubernetes.io/projected/9a986cc4-88d7-432a-8cc8-f2d5371373b1-kube-api-access-7fr6f\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:44.091227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.089476 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-token\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:44.091227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.089644 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-data\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:44.091227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.089676 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-workload-certs\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:44.091227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.089746 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istiod-ca-cert\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:44.091227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.089788 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-envoy\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:44.091227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.089843 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-credential-socket\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:44.091227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.089885 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-podinfo\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:44.091227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.089943 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-workload-socket\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:44.091227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.090510 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-workload-socket\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:44.130230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.093023 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-data\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:44.130230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.093325 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-workload-certs\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:44.130230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.093936 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istiod-ca-cert\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:44.130230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.096972 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-credential-socket\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:44.158726 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.156017 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-podinfo\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:44.158726 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.157646 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-envoy\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:44.195226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.195116 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fr6f\" (UniqueName: \"kubernetes.io/projected/9a986cc4-88d7-432a-8cc8-f2d5371373b1-kube-api-access-7fr6f\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:44.229335 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.228773 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltcdg\" (UniqueName: \"kubernetes.io/projected/1bc97146-d7dd-47a6-8353-ec71a36073ac-kube-api-access-ltcdg\") pod \"test-curl-7599-15b816\" (UID: \"1bc97146-d7dd-47a6-8353-ec71a36073ac\") " pod="opendatahub/test-curl-7599-15b816" Oct 29 23:17:44.355788 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.355758 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-token\") pod \"e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:44.361816 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.360752 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltcdg\" (UniqueName: \"kubernetes.io/projected/1bc97146-d7dd-47a6-8353-ec71a36073ac-kube-api-access-ltcdg\") pod \"test-curl-7599-15b816\" (UID: \"1bc97146-d7dd-47a6-8353-ec71a36073ac\") " pod="opendatahub/test-curl-7599-15b816" Oct 29 23:17:44.385498 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.385474 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltcdg\" (UniqueName: \"kubernetes.io/projected/1bc97146-d7dd-47a6-8353-ec71a36073ac-kube-api-access-ltcdg\") pod \"test-curl-7599-15b816\" (UID: \"1bc97146-d7dd-47a6-8353-ec71a36073ac\") " pod="opendatahub/test-curl-7599-15b816" Oct 29 23:17:44.396595 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.396566 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-15b816" Oct 29 23:17:44.582895 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.582839 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7599-ce7135" podUID="4c9b5f23-a4b2-4d19-83c3-43e29f69071a" containerName="test-curl-7599-ce7135" containerID="cri-o://60a27500d456d90ea6ca235c8b4db6625b68cf4c8ce5f232da07d3aaf97c5878" gracePeriod=2 Oct 29 23:17:44.585118 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.585091 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:44.754393 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.753687 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7599-15b816"] Oct 29 23:17:44.756263 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:17:44.755980 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bc97146_d7dd_47a6_8353_ec71a36073ac.slice/crio-b30c878fde7fbbfa17dc031fc32f9b8bb27df31199c5e448f564847d3d3cc587 WatchSource:0}: Error finding container b30c878fde7fbbfa17dc031fc32f9b8bb27df31199c5e448f564847d3d3cc587: Status 404 returned error can't find the container with id b30c878fde7fbbfa17dc031fc32f9b8bb27df31199c5e448f564847d3d3cc587 Oct 29 23:17:44.950461 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.950427 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f"] Oct 29 23:17:44.951383 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:17:44.951360 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a986cc4_88d7_432a_8cc8_f2d5371373b1.slice/crio-19454a3fc92fd8912b38a0c83675e18bdb5db6e0d03940e0191a546200cbecdf WatchSource:0}: Error finding container 19454a3fc92fd8912b38a0c83675e18bdb5db6e0d03940e0191a546200cbecdf: Status 404 returned error can't find the container with id 19454a3fc92fd8912b38a0c83675e18bdb5db6e0d03940e0191a546200cbecdf Oct 29 23:17:44.957066 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.957032 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:17:44.957159 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.957101 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:17:44.957159 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:44.957127 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:17:45.594412 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:45.594376 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" event={"ID":"9a986cc4-88d7-432a-8cc8-f2d5371373b1","Type":"ContainerStarted","Data":"c1cdf8455f07181d0ffbed0d991db9b77bbcef63bb87b23087abf6a61b0404ef"} Oct 29 23:17:45.594991 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:45.594964 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" event={"ID":"9a986cc4-88d7-432a-8cc8-f2d5371373b1","Type":"ContainerStarted","Data":"19454a3fc92fd8912b38a0c83675e18bdb5db6e0d03940e0191a546200cbecdf"} Oct 29 23:17:45.596351 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:45.596327 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7599-15b816" event={"ID":"1bc97146-d7dd-47a6-8353-ec71a36073ac","Type":"ContainerStarted","Data":"4493afdf75b8e2e004fa73690b5706e69dceff2c12177cd0510ae3932dc10930"} Oct 29 23:17:45.596455 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:45.596358 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7599-15b816" event={"ID":"1bc97146-d7dd-47a6-8353-ec71a36073ac","Type":"ContainerStarted","Data":"b30c878fde7fbbfa17dc031fc32f9b8bb27df31199c5e448f564847d3d3cc587"} Oct 29 23:17:45.644195 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:45.644119 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" podStartSLOduration=2.644096386 podStartE2EDuration="2.644096386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:45.641742042 +0000 UTC m=+2119.582289954" watchObservedRunningTime="2026-10-29 23:17:45.644096386 +0000 UTC m=+2119.584644300" Oct 29 23:17:45.692102 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:45.692039 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7599-15b816" podStartSLOduration=2.692020996 podStartE2EDuration="2.692020996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:45.684707443 +0000 UTC m=+2119.625255352" watchObservedRunningTime="2026-10-29 23:17:45.692020996 +0000 UTC m=+2119.632568907" Oct 29 23:17:46.585561 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:46.585532 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:46.592747 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:46.592721 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:46.619735 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:46.619705 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:46.620757 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:46.620736 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:17:46.647955 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:46.647928 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7599-15b816"] Oct 29 23:17:46.670664 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:46.670629 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7599-15b816"] Oct 29 23:17:46.807160 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:46.807126 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7599-201947"] Oct 29 23:17:46.807912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:46.807892 4821 cpu_manager.go:414] "containerMap: removing container" podUID="1bc97146-d7dd-47a6-8353-ec71a36073ac" containerName="test-curl-7599-15b816" Oct 29 23:17:46.808067 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:46.808042 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc97146-d7dd-47a6-8353-ec71a36073ac" containerName="test-curl-7599-15b816" Oct 29 23:17:46.808298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:46.808282 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bc97146-d7dd-47a6-8353-ec71a36073ac" containerName="test-curl-7599-15b816" Oct 29 23:17:46.829804 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:46.829769 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-201947" Oct 29 23:17:46.832185 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:46.832149 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7599-201947"] Oct 29 23:17:46.870458 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:46.870429 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-ce7135" Oct 29 23:17:46.923746 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:46.923713 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c9b5f23-a4b2-4d19-83c3-43e29f69071a-kube-api-access-6zr2j\" (UniqueName: \"kubernetes.io/projected/4c9b5f23-a4b2-4d19-83c3-43e29f69071a-kube-api-access-6zr2j\") pod \"4c9b5f23-a4b2-4d19-83c3-43e29f69071a\" (UID: \"4c9b5f23-a4b2-4d19-83c3-43e29f69071a\") " Oct 29 23:17:46.924301 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:46.924270 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m86v5\" (UniqueName: \"kubernetes.io/projected/a058d052-e845-42a4-beff-67fe02930cd0-kube-api-access-m86v5\") pod \"test-curl-7599-201947\" (UID: \"a058d052-e845-42a4-beff-67fe02930cd0\") " pod="opendatahub/test-curl-7599-201947" Oct 29 23:17:46.926104 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:46.926077 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9b5f23-a4b2-4d19-83c3-43e29f69071a-kube-api-access-6zr2j" pod "4c9b5f23-a4b2-4d19-83c3-43e29f69071a" (UID: "4c9b5f23-a4b2-4d19-83c3-43e29f69071a"). InnerVolumeSpecName "kube-api-access-6zr2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:17:47.026472 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:47.026437 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m86v5\" (UniqueName: \"kubernetes.io/projected/a058d052-e845-42a4-beff-67fe02930cd0-kube-api-access-m86v5\") pod \"test-curl-7599-201947\" (UID: \"a058d052-e845-42a4-beff-67fe02930cd0\") " pod="opendatahub/test-curl-7599-201947" Oct 29 23:17:47.026975 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:47.026953 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zr2j\" (UniqueName: \"kubernetes.io/projected/4c9b5f23-a4b2-4d19-83c3-43e29f69071a-kube-api-access-6zr2j\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:17:47.057737 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:47.057703 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m86v5\" (UniqueName: \"kubernetes.io/projected/a058d052-e845-42a4-beff-67fe02930cd0-kube-api-access-m86v5\") pod \"test-curl-7599-201947\" (UID: \"a058d052-e845-42a4-beff-67fe02930cd0\") " pod="opendatahub/test-curl-7599-201947" Oct 29 23:17:47.155523 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:47.155482 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-201947" Oct 29 23:17:47.476018 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:47.475987 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7599-201947"] Oct 29 23:17:47.477129 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:17:47.477094 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda058d052_e845_42a4_beff_67fe02930cd0.slice/crio-f5f849a839940a7ecbbe7a42a2463073952c9e5516d607fad40a5831c7840b6a WatchSource:0}: Error finding container f5f849a839940a7ecbbe7a42a2463073952c9e5516d607fad40a5831c7840b6a: Status 404 returned error can't find the container with id f5f849a839940a7ecbbe7a42a2463073952c9e5516d607fad40a5831c7840b6a Oct 29 23:17:47.630604 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:47.630566 4821 generic.go:356] "Generic (PLEG): container finished" podID="4c9b5f23-a4b2-4d19-83c3-43e29f69071a" containerID="60a27500d456d90ea6ca235c8b4db6625b68cf4c8ce5f232da07d3aaf97c5878" exitCode=137 Oct 29 23:17:47.631073 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:47.630636 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-ce7135" Oct 29 23:17:47.631073 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:47.630665 4821 scope.go:122] "RemoveContainer" containerID="60a27500d456d90ea6ca235c8b4db6625b68cf4c8ce5f232da07d3aaf97c5878" Oct 29 23:17:47.633767 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:47.632874 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7599-15b816" podUID="1bc97146-d7dd-47a6-8353-ec71a36073ac" containerName="test-curl-7599-15b816" containerID="cri-o://4493afdf75b8e2e004fa73690b5706e69dceff2c12177cd0510ae3932dc10930" gracePeriod=2 Oct 29 23:17:47.633865 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:47.633817 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7599-201947" event={"ID":"a058d052-e845-42a4-beff-67fe02930cd0","Type":"ContainerStarted","Data":"ad546868bdfe1baf7948f5999a908f42a0a234b5ec34860cd6eb96741fc7e92f"} Oct 29 23:17:47.633865 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:47.633859 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7599-201947" event={"ID":"a058d052-e845-42a4-beff-67fe02930cd0","Type":"ContainerStarted","Data":"f5f849a839940a7ecbbe7a42a2463073952c9e5516d607fad40a5831c7840b6a"} Oct 29 23:17:47.648949 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:47.648924 4821 scope.go:122] "RemoveContainer" containerID="60a27500d456d90ea6ca235c8b4db6625b68cf4c8ce5f232da07d3aaf97c5878" Oct 29 23:17:47.650531 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:17:47.650491 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60a27500d456d90ea6ca235c8b4db6625b68cf4c8ce5f232da07d3aaf97c5878\": container with ID starting with 60a27500d456d90ea6ca235c8b4db6625b68cf4c8ce5f232da07d3aaf97c5878 not found: ID does not exist" containerID="60a27500d456d90ea6ca235c8b4db6625b68cf4c8ce5f232da07d3aaf97c5878" Oct 29 23:17:47.650650 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:47.650533 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60a27500d456d90ea6ca235c8b4db6625b68cf4c8ce5f232da07d3aaf97c5878"} err="failed to get container status \"60a27500d456d90ea6ca235c8b4db6625b68cf4c8ce5f232da07d3aaf97c5878\": rpc error: code = NotFound desc = could not find container \"60a27500d456d90ea6ca235c8b4db6625b68cf4c8ce5f232da07d3aaf97c5878\": container with ID starting with 60a27500d456d90ea6ca235c8b4db6625b68cf4c8ce5f232da07d3aaf97c5878 not found: ID does not exist" Oct 29 23:17:47.679154 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:47.679097 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7599-201947" podStartSLOduration=1.679084058 podStartE2EDuration="1.679084058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:47.675855744 +0000 UTC m=+2121.616403655" watchObservedRunningTime="2026-10-29 23:17:47.679084058 +0000 UTC m=+2121.619631967" Oct 29 23:17:47.691129 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:17:47.691087 4821 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7599-15b816\" is forbidden: User \"system:node:ip-10-0-130-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-252.ec2.internal' and this object" podUID="1bc97146-d7dd-47a6-8353-ec71a36073ac" pod="opendatahub/test-curl-7599-15b816" Oct 29 23:17:47.705755 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:17:47.705715 4821 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7599-ce7135\" is forbidden: User \"system:node:ip-10-0-130-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-252.ec2.internal' and this object" podUID="4c9b5f23-a4b2-4d19-83c3-43e29f69071a" pod="opendatahub/test-curl-7599-ce7135" Oct 29 23:17:47.720296 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:17:47.720253 4821 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7599-15b816\" is forbidden: User \"system:node:ip-10-0-130-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-252.ec2.internal' and this object" podUID="1bc97146-d7dd-47a6-8353-ec71a36073ac" pod="opendatahub/test-curl-7599-15b816" Oct 29 23:17:48.172069 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:48.172037 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c9b5f23-a4b2-4d19-83c3-43e29f69071a" path="/var/lib/kubelet/pods/4c9b5f23-a4b2-4d19-83c3-43e29f69071a/volumes" Oct 29 23:17:48.563021 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:48.562936 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7599-201947"] Oct 29 23:17:48.571099 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:48.571064 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7599-201947"] Oct 29 23:17:48.729967 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:48.729924 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7599-5e0bb7"] Oct 29 23:17:48.730899 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:48.730874 4821 cpu_manager.go:414] "containerMap: removing container" podUID="a058d052-e845-42a4-beff-67fe02930cd0" containerName="test-curl-7599-201947" Oct 29 23:17:48.730899 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:48.730897 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a058d052-e845-42a4-beff-67fe02930cd0" containerName="test-curl-7599-201947" Oct 29 23:17:48.731134 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:48.731118 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="a058d052-e845-42a4-beff-67fe02930cd0" containerName="test-curl-7599-201947" Oct 29 23:17:48.748804 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:48.748767 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-5e0bb7" Oct 29 23:17:48.754784 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:48.754729 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7599-5e0bb7"] Oct 29 23:17:48.858012 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:48.857978 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqngx\" (UniqueName: \"kubernetes.io/projected/530158a0-8ef5-4adc-933e-50850e166c0d-kube-api-access-mqngx\") pod \"test-curl-7599-5e0bb7\" (UID: \"530158a0-8ef5-4adc-933e-50850e166c0d\") " pod="opendatahub/test-curl-7599-5e0bb7" Oct 29 23:17:48.959857 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:48.959807 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqngx\" (UniqueName: \"kubernetes.io/projected/530158a0-8ef5-4adc-933e-50850e166c0d-kube-api-access-mqngx\") pod \"test-curl-7599-5e0bb7\" (UID: \"530158a0-8ef5-4adc-933e-50850e166c0d\") " pod="opendatahub/test-curl-7599-5e0bb7" Oct 29 23:17:48.984134 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:48.984103 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqngx\" (UniqueName: \"kubernetes.io/projected/530158a0-8ef5-4adc-933e-50850e166c0d-kube-api-access-mqngx\") pod \"test-curl-7599-5e0bb7\" (UID: \"530158a0-8ef5-4adc-933e-50850e166c0d\") " pod="opendatahub/test-curl-7599-5e0bb7" Oct 29 23:17:49.076222 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:49.074126 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-5e0bb7" Oct 29 23:17:49.682073 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:49.680119 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7599-201947" podUID="a058d052-e845-42a4-beff-67fe02930cd0" containerName="test-curl-7599-201947" containerID="cri-o://ad546868bdfe1baf7948f5999a908f42a0a234b5ec34860cd6eb96741fc7e92f" gracePeriod=2 Oct 29 23:17:49.695135 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:17:49.695100 4821 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7599-201947\" is forbidden: User \"system:node:ip-10-0-130-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-252.ec2.internal' and this object" podUID="a058d052-e845-42a4-beff-67fe02930cd0" pod="opendatahub/test-curl-7599-201947" Oct 29 23:17:49.859290 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:49.859258 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7599-5e0bb7"] Oct 29 23:17:49.859947 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:17:49.859918 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod530158a0_8ef5_4adc_933e_50850e166c0d.slice/crio-6f61fa3e035052b98a4eb18bde05540b0a67c0ab2e6c9f8685f84364b7e8653b WatchSource:0}: Error finding container 6f61fa3e035052b98a4eb18bde05540b0a67c0ab2e6c9f8685f84364b7e8653b: Status 404 returned error can't find the container with id 6f61fa3e035052b98a4eb18bde05540b0a67c0ab2e6c9f8685f84364b7e8653b Oct 29 23:17:49.964802 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:49.964772 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-15b816" Oct 29 23:17:49.979675 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:17:49.979641 4821 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7599-201947\" is forbidden: User \"system:node:ip-10-0-130-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-252.ec2.internal' and this object" podUID="a058d052-e845-42a4-beff-67fe02930cd0" pod="opendatahub/test-curl-7599-201947" Oct 29 23:17:50.020413 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:50.020377 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bc97146-d7dd-47a6-8353-ec71a36073ac-kube-api-access-ltcdg\" (UniqueName: \"kubernetes.io/projected/1bc97146-d7dd-47a6-8353-ec71a36073ac-kube-api-access-ltcdg\") pod \"1bc97146-d7dd-47a6-8353-ec71a36073ac\" (UID: \"1bc97146-d7dd-47a6-8353-ec71a36073ac\") " Oct 29 23:17:50.110806 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:50.109737 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc97146-d7dd-47a6-8353-ec71a36073ac-kube-api-access-ltcdg" pod "1bc97146-d7dd-47a6-8353-ec71a36073ac" (UID: "1bc97146-d7dd-47a6-8353-ec71a36073ac"). InnerVolumeSpecName "kube-api-access-ltcdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:17:50.125669 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:50.122934 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltcdg\" (UniqueName: \"kubernetes.io/projected/1bc97146-d7dd-47a6-8353-ec71a36073ac-kube-api-access-ltcdg\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:17:50.306081 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:50.305071 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bc97146-d7dd-47a6-8353-ec71a36073ac" path="/var/lib/kubelet/pods/1bc97146-d7dd-47a6-8353-ec71a36073ac/volumes" Oct 29 23:17:50.688553 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:50.688517 4821 generic.go:356] "Generic (PLEG): container finished" podID="1bc97146-d7dd-47a6-8353-ec71a36073ac" containerID="4493afdf75b8e2e004fa73690b5706e69dceff2c12177cd0510ae3932dc10930" exitCode=137 Oct 29 23:17:50.689440 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:50.688567 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-15b816" Oct 29 23:17:50.689440 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:50.688606 4821 scope.go:122] "RemoveContainer" containerID="4493afdf75b8e2e004fa73690b5706e69dceff2c12177cd0510ae3932dc10930" Oct 29 23:17:50.691518 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:50.691475 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7599-5e0bb7" event={"ID":"530158a0-8ef5-4adc-933e-50850e166c0d","Type":"ContainerStarted","Data":"b243cab7e0048d90b7c7bf1d04d4d3ad81b506711d31d8028c32e14682ab3f0e"} Oct 29 23:17:50.691650 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:50.691522 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7599-5e0bb7" event={"ID":"530158a0-8ef5-4adc-933e-50850e166c0d","Type":"ContainerStarted","Data":"6f61fa3e035052b98a4eb18bde05540b0a67c0ab2e6c9f8685f84364b7e8653b"} Oct 29 23:17:50.700449 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:50.700427 4821 scope.go:122] "RemoveContainer" containerID="4493afdf75b8e2e004fa73690b5706e69dceff2c12177cd0510ae3932dc10930" Oct 29 23:17:50.700715 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:17:50.700685 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4493afdf75b8e2e004fa73690b5706e69dceff2c12177cd0510ae3932dc10930\": container with ID starting with 4493afdf75b8e2e004fa73690b5706e69dceff2c12177cd0510ae3932dc10930 not found: ID does not exist" containerID="4493afdf75b8e2e004fa73690b5706e69dceff2c12177cd0510ae3932dc10930" Oct 29 23:17:50.700813 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:50.700714 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4493afdf75b8e2e004fa73690b5706e69dceff2c12177cd0510ae3932dc10930"} err="failed to get container status \"4493afdf75b8e2e004fa73690b5706e69dceff2c12177cd0510ae3932dc10930\": rpc error: code = NotFound desc = could not find container \"4493afdf75b8e2e004fa73690b5706e69dceff2c12177cd0510ae3932dc10930\": container with ID starting with 4493afdf75b8e2e004fa73690b5706e69dceff2c12177cd0510ae3932dc10930 not found: ID does not exist" Oct 29 23:17:50.732415 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:50.732329 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7599-5e0bb7" podStartSLOduration=2.73231286 podStartE2EDuration="2.73231286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:50.730966502 +0000 UTC m=+2124.671514412" watchObservedRunningTime="2026-10-29 23:17:50.73231286 +0000 UTC m=+2124.672860772" Oct 29 23:17:51.187982 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:51.187944 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 29 23:17:51.188467 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:51.188022 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" podUID="36ea1460-ac2f-410a-b06a-b18f1f8e56d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 29 23:17:51.660219 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:51.652253 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7599-5e0bb7"] Oct 29 23:17:51.667170 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:51.667119 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7599-5e0bb7"] Oct 29 23:17:51.705331 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:51.705293 4821 generic.go:356] "Generic (PLEG): container finished" podID="36ea1460-ac2f-410a-b06a-b18f1f8e56d3" containerID="528e90721b9ce1aad59011a013291ebbf49717ddb34cf6e3b7c39895fb10beb7" exitCode=0 Oct 29 23:17:51.705493 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:51.705362 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" event={"ID":"36ea1460-ac2f-410a-b06a-b18f1f8e56d3","Type":"ContainerDied","Data":"528e90721b9ce1aad59011a013291ebbf49717ddb34cf6e3b7c39895fb10beb7"} Oct 29 23:17:51.705943 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:51.705919 4821 scope.go:122] "RemoveContainer" containerID="528e90721b9ce1aad59011a013291ebbf49717ddb34cf6e3b7c39895fb10beb7" Oct 29 23:17:51.708978 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:51.708942 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:17:51.709107 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:51.709037 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:17:51.709107 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:51.709092 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:17:52.042718 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.042694 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-201947" Oct 29 23:17:52.081397 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.081367 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a058d052-e845-42a4-beff-67fe02930cd0-kube-api-access-m86v5\" (UniqueName: \"kubernetes.io/projected/a058d052-e845-42a4-beff-67fe02930cd0-kube-api-access-m86v5\") pod \"a058d052-e845-42a4-beff-67fe02930cd0\" (UID: \"a058d052-e845-42a4-beff-67fe02930cd0\") " Oct 29 23:17:52.083896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.083791 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a058d052-e845-42a4-beff-67fe02930cd0-kube-api-access-m86v5" pod "a058d052-e845-42a4-beff-67fe02930cd0" (UID: "a058d052-e845-42a4-beff-67fe02930cd0"). InnerVolumeSpecName "kube-api-access-m86v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:17:52.173396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.173363 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a058d052-e845-42a4-beff-67fe02930cd0" path="/var/lib/kubelet/pods/a058d052-e845-42a4-beff-67fe02930cd0/volumes" Oct 29 23:17:52.183883 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.183857 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m86v5\" (UniqueName: \"kubernetes.io/projected/a058d052-e845-42a4-beff-67fe02930cd0-kube-api-access-m86v5\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:17:52.584595 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.584506 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd"] Oct 29 23:17:52.585680 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.585655 4821 cpu_manager.go:414] "containerMap: removing container" podUID="530158a0-8ef5-4adc-933e-50850e166c0d" containerName="test-curl-7599-5e0bb7" Oct 29 23:17:52.585805 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.585683 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530158a0-8ef5-4adc-933e-50850e166c0d" containerName="test-curl-7599-5e0bb7" Oct 29 23:17:52.585948 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.585932 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="530158a0-8ef5-4adc-933e-50850e166c0d" containerName="test-curl-7599-5e0bb7" Oct 29 23:17:52.591532 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.590729 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.641712 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.641682 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-dcacb9-openshift-default-dockercfg-4g4gh\"" Oct 29 23:17:52.686740 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.686690 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd"] Oct 29 23:17:52.693984 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.693096 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-credential-socket\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.693984 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.693240 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-envoy\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.693984 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.693399 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk72l\" (UniqueName: \"kubernetes.io/projected/d560c3fe-b142-4ae7-a7da-c86260287c18-kube-api-access-dk72l\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.693984 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.693598 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-data\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.693984 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.693628 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-workload-certs\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.693984 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.693743 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-podinfo\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.693984 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.693854 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d560c3fe-b142-4ae7-a7da-c86260287c18-istiod-ca-cert\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.693984 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.693892 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-token\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.693984 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.693917 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-workload-socket\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.752018 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.751979 4821 generic.go:356] "Generic (PLEG): container finished" podID="a058d052-e845-42a4-beff-67fe02930cd0" containerID="ad546868bdfe1baf7948f5999a908f42a0a234b5ec34860cd6eb96741fc7e92f" exitCode=137 Oct 29 23:17:52.752702 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.752646 4821 scope.go:122] "RemoveContainer" containerID="ad546868bdfe1baf7948f5999a908f42a0a234b5ec34860cd6eb96741fc7e92f" Oct 29 23:17:52.753062 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.753045 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-201947" Oct 29 23:17:52.759329 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.759292 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" event={"ID":"36ea1460-ac2f-410a-b06a-b18f1f8e56d3","Type":"ContainerStarted","Data":"53156b884a4309f21bd321f88c0fc944f19c04abf6df11ae89d29f3177f85736"} Oct 29 23:17:52.759598 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.759550 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7599-5e0bb7" podUID="530158a0-8ef5-4adc-933e-50850e166c0d" containerName="test-curl-7599-5e0bb7" containerID="cri-o://b243cab7e0048d90b7c7bf1d04d4d3ad81b506711d31d8028c32e14682ab3f0e" gracePeriod=2 Oct 29 23:17:52.770344 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.770325 4821 scope.go:122] "RemoveContainer" containerID="ad546868bdfe1baf7948f5999a908f42a0a234b5ec34860cd6eb96741fc7e92f" Oct 29 23:17:52.770649 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:17:52.770623 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad546868bdfe1baf7948f5999a908f42a0a234b5ec34860cd6eb96741fc7e92f\": container with ID starting with ad546868bdfe1baf7948f5999a908f42a0a234b5ec34860cd6eb96741fc7e92f not found: ID does not exist" containerID="ad546868bdfe1baf7948f5999a908f42a0a234b5ec34860cd6eb96741fc7e92f" Oct 29 23:17:52.770751 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.770692 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad546868bdfe1baf7948f5999a908f42a0a234b5ec34860cd6eb96741fc7e92f"} err="failed to get container status \"ad546868bdfe1baf7948f5999a908f42a0a234b5ec34860cd6eb96741fc7e92f\": rpc error: code = NotFound desc = could not find container \"ad546868bdfe1baf7948f5999a908f42a0a234b5ec34860cd6eb96741fc7e92f\": container with ID starting with ad546868bdfe1baf7948f5999a908f42a0a234b5ec34860cd6eb96741fc7e92f not found: ID does not exist" Oct 29 23:17:52.795223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.795164 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d560c3fe-b142-4ae7-a7da-c86260287c18-istiod-ca-cert\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.795390 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.795261 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-token\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.795390 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.795286 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-workload-socket\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.795390 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.795324 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-credential-socket\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.795390 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.795391 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-envoy\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.795660 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.795425 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk72l\" (UniqueName: \"kubernetes.io/projected/d560c3fe-b142-4ae7-a7da-c86260287c18-kube-api-access-dk72l\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.795660 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.795523 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-data\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.795660 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.795545 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-workload-certs\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.795660 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.795625 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-podinfo\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.796312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.796286 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-workload-socket\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.796542 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.796522 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-credential-socket\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.796652 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.796571 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d560c3fe-b142-4ae7-a7da-c86260287c18-istiod-ca-cert\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.796865 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.796838 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-data\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.797025 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.797002 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-workload-certs\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.800165 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.799940 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-podinfo\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.800345 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.800168 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-envoy\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.861276 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.859957 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk72l\" (UniqueName: \"kubernetes.io/projected/d560c3fe-b142-4ae7-a7da-c86260287c18-kube-api-access-dk72l\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.861276 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.860389 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-token\") pod \"e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:52.928568 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:52.928525 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:53.070352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:53.066576 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:17:53.070352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:53.066721 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 29 23:17:53.070352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:53.066781 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" podUID="36ea1460-ac2f-410a-b06a-b18f1f8e56d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 29 23:17:53.668448 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:53.668421 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd"] Oct 29 23:17:53.669698 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:17:53.669666 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd560c3fe_b142_4ae7_a7da_c86260287c18.slice/crio-c31dba572c3a649d70073d9bb9f0769931652f36705c9d3ecd411e6e23a926ec WatchSource:0}: Error finding container c31dba572c3a649d70073d9bb9f0769931652f36705c9d3ecd411e6e23a926ec: Status 404 returned error can't find the container with id c31dba572c3a649d70073d9bb9f0769931652f36705c9d3ecd411e6e23a926ec Oct 29 23:17:53.673422 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:53.673377 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:17:53.673523 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:53.673474 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:17:53.673570 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:53.673518 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:17:53.773700 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:53.773665 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" event={"ID":"d560c3fe-b142-4ae7-a7da-c86260287c18","Type":"ContainerStarted","Data":"c31dba572c3a649d70073d9bb9f0769931652f36705c9d3ecd411e6e23a926ec"} Oct 29 23:17:54.011273 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:17:54.009455 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:17:54.066559 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:54.066528 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 29 23:17:54.066837 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:54.066811 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" podUID="36ea1460-ac2f-410a-b06a-b18f1f8e56d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 29 23:17:54.786554 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:54.786515 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" event={"ID":"d560c3fe-b142-4ae7-a7da-c86260287c18","Type":"ContainerStarted","Data":"9c9a20c330af433a30b361d363749ffd131ec1fd73f3cd04bfe058a5c83d6199"} Oct 29 23:17:54.855620 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:54.855556 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" podStartSLOduration=2.8555384569999998 podStartE2EDuration="2.855538457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:17:54.848905474 +0000 UTC m=+2128.789453385" watchObservedRunningTime="2026-10-29 23:17:54.855538457 +0000 UTC m=+2128.796086369" Oct 29 23:17:54.961228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:54.959973 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:55.089287 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:55.077773 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:55.095538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:55.095503 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 29 23:17:55.095767 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:55.095741 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" podUID="36ea1460-ac2f-410a-b06a-b18f1f8e56d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 29 23:17:55.200031 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:55.199994 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-5e0bb7" Oct 29 23:17:55.254011 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:55.253974 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530158a0-8ef5-4adc-933e-50850e166c0d-kube-api-access-mqngx\" (UniqueName: \"kubernetes.io/projected/530158a0-8ef5-4adc-933e-50850e166c0d-kube-api-access-mqngx\") pod \"530158a0-8ef5-4adc-933e-50850e166c0d\" (UID: \"530158a0-8ef5-4adc-933e-50850e166c0d\") " Oct 29 23:17:55.257794 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:55.257759 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530158a0-8ef5-4adc-933e-50850e166c0d-kube-api-access-mqngx" pod "530158a0-8ef5-4adc-933e-50850e166c0d" (UID: "530158a0-8ef5-4adc-933e-50850e166c0d"). InnerVolumeSpecName "kube-api-access-mqngx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:17:55.356912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:55.356881 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqngx\" (UniqueName: \"kubernetes.io/projected/530158a0-8ef5-4adc-933e-50850e166c0d-kube-api-access-mqngx\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:17:55.799653 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:55.799555 4821 generic.go:356] "Generic (PLEG): container finished" podID="530158a0-8ef5-4adc-933e-50850e166c0d" containerID="b243cab7e0048d90b7c7bf1d04d4d3ad81b506711d31d8028c32e14682ab3f0e" exitCode=137 Oct 29 23:17:55.799653 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:55.799598 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7599-5e0bb7" Oct 29 23:17:55.800178 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:55.799652 4821 scope.go:122] "RemoveContainer" containerID="b243cab7e0048d90b7c7bf1d04d4d3ad81b506711d31d8028c32e14682ab3f0e" Oct 29 23:17:55.800271 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:55.800238 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:55.801569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:55.801551 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:17:55.811384 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:55.811361 4821 scope.go:122] "RemoveContainer" containerID="b243cab7e0048d90b7c7bf1d04d4d3ad81b506711d31d8028c32e14682ab3f0e" Oct 29 23:17:55.811661 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:17:55.811643 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b243cab7e0048d90b7c7bf1d04d4d3ad81b506711d31d8028c32e14682ab3f0e\": container with ID starting with b243cab7e0048d90b7c7bf1d04d4d3ad81b506711d31d8028c32e14682ab3f0e not found: ID does not exist" containerID="b243cab7e0048d90b7c7bf1d04d4d3ad81b506711d31d8028c32e14682ab3f0e" Oct 29 23:17:55.811707 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:55.811669 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b243cab7e0048d90b7c7bf1d04d4d3ad81b506711d31d8028c32e14682ab3f0e"} err="failed to get container status \"b243cab7e0048d90b7c7bf1d04d4d3ad81b506711d31d8028c32e14682ab3f0e\": rpc error: code = NotFound desc = could not find container \"b243cab7e0048d90b7c7bf1d04d4d3ad81b506711d31d8028c32e14682ab3f0e\": container with ID starting with b243cab7e0048d90b7c7bf1d04d4d3ad81b506711d31d8028c32e14682ab3f0e not found: ID does not exist" Oct 29 23:17:55.894754 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:17:55.894713 4821 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7599-5e0bb7\" is forbidden: User \"system:node:ip-10-0-130-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-252.ec2.internal' and this object" podUID="530158a0-8ef5-4adc-933e-50850e166c0d" pod="opendatahub/test-curl-7599-5e0bb7" Oct 29 23:17:56.066912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:56.066822 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 29 23:17:56.066912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:56.066878 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" podUID="36ea1460-ac2f-410a-b06a-b18f1f8e56d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 29 23:17:56.219560 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:17:56.219517 4821 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7599-5e0bb7\" is forbidden: User \"system:node:ip-10-0-130-252.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-252.ec2.internal' and this object" podUID="530158a0-8ef5-4adc-933e-50850e166c0d" pod="opendatahub/test-curl-7599-5e0bb7" Oct 29 23:17:56.306745 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:56.247048 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="530158a0-8ef5-4adc-933e-50850e166c0d" path="/var/lib/kubelet/pods/530158a0-8ef5-4adc-933e-50850e166c0d/volumes" Oct 29 23:17:57.066311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:57.066271 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 29 23:17:57.066784 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:57.066333 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" podUID="36ea1460-ac2f-410a-b06a-b18f1f8e56d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 29 23:17:58.066962 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:58.066915 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 29 23:17:58.067427 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:58.066986 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" podUID="36ea1460-ac2f-410a-b06a-b18f1f8e56d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 29 23:17:59.071912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:59.071881 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:17:59.072464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:59.072440 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:17:59.073087 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:17:59.073067 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:18:10.990130 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:10.990094 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn"] Oct 29 23:18:11.005151 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:11.005126 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn" Oct 29 23:18:11.023054 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:11.023012 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn"] Oct 29 23:18:11.030284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:11.030256 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b2b71e-serving-cert\"" Oct 29 23:18:11.094081 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:11.094039 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8314c4e0-da83-4173-bc1a-7fb974a7ab36-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn\" (UID: \"8314c4e0-da83-4173-bc1a-7fb974a7ab36\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn" Oct 29 23:18:11.094298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:11.094190 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8314c4e0-da83-4173-bc1a-7fb974a7ab36-metrics-certs\") pod \"maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn\" (UID: \"8314c4e0-da83-4173-bc1a-7fb974a7ab36\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn" Oct 29 23:18:11.094357 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:11.094305 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwbv5\" (UniqueName: \"kubernetes.io/projected/8314c4e0-da83-4173-bc1a-7fb974a7ab36-kube-api-access-zwbv5\") pod \"maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn\" (UID: \"8314c4e0-da83-4173-bc1a-7fb974a7ab36\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn" Oct 29 23:18:11.094462 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:11.094445 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8314c4e0-da83-4173-bc1a-7fb974a7ab36-maas-api-tls\") pod \"maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn\" (UID: \"8314c4e0-da83-4173-bc1a-7fb974a7ab36\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn" Oct 29 23:18:11.196522 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:11.196480 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8314c4e0-da83-4173-bc1a-7fb974a7ab36-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn\" (UID: \"8314c4e0-da83-4173-bc1a-7fb974a7ab36\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn" Oct 29 23:18:11.196753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:11.196574 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8314c4e0-da83-4173-bc1a-7fb974a7ab36-metrics-certs\") pod \"maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn\" (UID: \"8314c4e0-da83-4173-bc1a-7fb974a7ab36\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn" Oct 29 23:18:11.196753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:11.196625 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwbv5\" (UniqueName: \"kubernetes.io/projected/8314c4e0-da83-4173-bc1a-7fb974a7ab36-kube-api-access-zwbv5\") pod \"maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn\" (UID: \"8314c4e0-da83-4173-bc1a-7fb974a7ab36\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn" Oct 29 23:18:11.196753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:11.196716 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8314c4e0-da83-4173-bc1a-7fb974a7ab36-maas-api-tls\") pod \"maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn\" (UID: \"8314c4e0-da83-4173-bc1a-7fb974a7ab36\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn" Oct 29 23:18:11.197273 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:11.197248 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8314c4e0-da83-4173-bc1a-7fb974a7ab36-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn\" (UID: \"8314c4e0-da83-4173-bc1a-7fb974a7ab36\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn" Oct 29 23:18:11.199238 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:11.199193 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8314c4e0-da83-4173-bc1a-7fb974a7ab36-metrics-certs\") pod \"maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn\" (UID: \"8314c4e0-da83-4173-bc1a-7fb974a7ab36\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn" Oct 29 23:18:11.199424 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:11.199405 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8314c4e0-da83-4173-bc1a-7fb974a7ab36-maas-api-tls\") pod \"maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn\" (UID: \"8314c4e0-da83-4173-bc1a-7fb974a7ab36\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn" Oct 29 23:18:11.223701 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:11.223673 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwbv5\" (UniqueName: \"kubernetes.io/projected/8314c4e0-da83-4173-bc1a-7fb974a7ab36-kube-api-access-zwbv5\") pod \"maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn\" (UID: \"8314c4e0-da83-4173-bc1a-7fb974a7ab36\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn" Oct 29 23:18:11.320062 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:11.319963 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn" Oct 29 23:18:11.648767 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:18:11.648736 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8314c4e0_da83_4173_bc1a_7fb974a7ab36.slice/crio-ca00e4942c41677de40d78468fda3a70ff375dfb94d9091bfb5e23f936977859 WatchSource:0}: Error finding container ca00e4942c41677de40d78468fda3a70ff375dfb94d9091bfb5e23f936977859: Status 404 returned error can't find the container with id ca00e4942c41677de40d78468fda3a70ff375dfb94d9091bfb5e23f936977859 Oct 29 23:18:11.649321 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:11.649296 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn"] Oct 29 23:18:11.923928 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:11.923813 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk"] Oct 29 23:18:11.924102 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:11.924062 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" podUID="2cdd98c2-c079-4880-9835-36d2c4a86de8" containerName="istio-proxy" containerID="cri-o://c9aae7a6235e709152958c1867747e4fa2f852bd6d7f2a2dabfc43b67fa20e64" gracePeriod=30 Oct 29 23:18:11.939784 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:11.939748 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd"] Oct 29 23:18:11.940237 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:11.940187 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" podUID="be674d35-b876-4f7d-9ceb-8cd285a3f2db" containerName="istio-proxy" containerID="cri-o://c2003f19cf7d6a4f05f4fb76fb6a106b857e911e3f85ecc5d14a785250bcf519" gracePeriod=30 Oct 29 23:18:12.134780 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:12.134718 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn" event={"ID":"8314c4e0-da83-4173-bc1a-7fb974a7ab36","Type":"ContainerStarted","Data":"5b3b7ae6ca738c3a5450687bad73204b7edb87cdc6a159a3a1c1d55917ea0c6c"} Oct 29 23:18:12.135257 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:12.134798 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn" event={"ID":"8314c4e0-da83-4173-bc1a-7fb974a7ab36","Type":"ContainerStarted","Data":"ca00e4942c41677de40d78468fda3a70ff375dfb94d9091bfb5e23f936977859"} Oct 29 23:18:12.135257 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:12.134832 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn" Oct 29 23:18:12.274304 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:12.259620 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn" podStartSLOduration=1.938255982 podStartE2EDuration="2.259594262s" totalImagesPullingTime="321.33828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:18:12.178660738 +0000 UTC m=+2146.119208649" watchObservedRunningTime="2026-10-29 23:18:12.259594262 +0000 UTC m=+2146.200142174" Oct 29 23:18:17.377081 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.377042 4821 generic.go:356] "Generic (PLEG): container finished" podID="be674d35-b876-4f7d-9ceb-8cd285a3f2db" containerID="c2003f19cf7d6a4f05f4fb76fb6a106b857e911e3f85ecc5d14a785250bcf519" exitCode=0 Oct 29 23:18:17.377589 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.377092 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" event={"ID":"be674d35-b876-4f7d-9ceb-8cd285a3f2db","Type":"ContainerDied","Data":"c2003f19cf7d6a4f05f4fb76fb6a106b857e911e3f85ecc5d14a785250bcf519"} Oct 29 23:18:17.380463 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.380432 4821 generic.go:356] "Generic (PLEG): container finished" podID="2cdd98c2-c079-4880-9835-36d2c4a86de8" containerID="c9aae7a6235e709152958c1867747e4fa2f852bd6d7f2a2dabfc43b67fa20e64" exitCode=0 Oct 29 23:18:17.380588 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.380568 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" event={"ID":"2cdd98c2-c079-4880-9835-36d2c4a86de8","Type":"ContainerDied","Data":"c9aae7a6235e709152958c1867747e4fa2f852bd6d7f2a2dabfc43b67fa20e64"} Oct 29 23:18:17.380659 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.380615 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" event={"ID":"2cdd98c2-c079-4880-9835-36d2c4a86de8","Type":"ContainerDied","Data":"df6cc09625ce9a4bff8800575225fcc9c26c0fac60f0821bdae35d6869ae3012"} Oct 29 23:18:17.380659 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.380642 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df6cc09625ce9a4bff8800575225fcc9c26c0fac60f0821bdae35d6869ae3012" Oct 29 23:18:17.406593 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.406566 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:18:17.455404 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.455369 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:18:17.532232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.526824 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-token\" (UniqueName: \"kubernetes.io/projected/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-token\") pod \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " Oct 29 23:18:17.532232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.526939 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cdd98c2-c079-4880-9835-36d2c4a86de8-kube-api-access-8gxdc\" (UniqueName: \"kubernetes.io/projected/2cdd98c2-c079-4880-9835-36d2c4a86de8-kube-api-access-8gxdc\") pod \"2cdd98c2-c079-4880-9835-36d2c4a86de8\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " Oct 29 23:18:17.532232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.526972 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-data\") pod \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " Oct 29 23:18:17.532232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.527001 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-workload-socket\") pod \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " Oct 29 23:18:17.532232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.527041 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istiod-ca-cert\") pod \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " Oct 29 23:18:17.532232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.527088 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-credential-socket\") pod \"2cdd98c2-c079-4880-9835-36d2c4a86de8\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " Oct 29 23:18:17.532232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.527186 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-workload-certs\") pod \"2cdd98c2-c079-4880-9835-36d2c4a86de8\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " Oct 29 23:18:17.532232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.527273 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-envoy\") pod \"2cdd98c2-c079-4880-9835-36d2c4a86de8\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " Oct 29 23:18:17.532232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.527318 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-credential-socket\") pod \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " Oct 29 23:18:17.532232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.527357 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cdd98c2-c079-4880-9835-36d2c4a86de8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdd98c2-c079-4880-9835-36d2c4a86de8-istiod-ca-cert\") pod \"2cdd98c2-c079-4880-9835-36d2c4a86de8\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " Oct 29 23:18:17.532232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.527424 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-data\") pod \"2cdd98c2-c079-4880-9835-36d2c4a86de8\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " Oct 29 23:18:17.532232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.527460 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-workload-socket\") pod \"2cdd98c2-c079-4880-9835-36d2c4a86de8\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " Oct 29 23:18:17.532232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.527512 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-envoy\") pod \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " Oct 29 23:18:17.532232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.527564 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be674d35-b876-4f7d-9ceb-8cd285a3f2db-kube-api-access-nfvlk\" (UniqueName: \"kubernetes.io/projected/be674d35-b876-4f7d-9ceb-8cd285a3f2db-kube-api-access-nfvlk\") pod \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " Oct 29 23:18:17.532232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.527623 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-token\" (UniqueName: \"kubernetes.io/projected/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-token\") pod \"2cdd98c2-c079-4880-9835-36d2c4a86de8\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " Oct 29 23:18:17.532232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.527702 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-podinfo\") pod \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " Oct 29 23:18:17.532232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.527806 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-workload-certs\") pod \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\" (UID: \"be674d35-b876-4f7d-9ceb-8cd285a3f2db\") " Oct 29 23:18:17.532232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.527900 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-podinfo\") pod \"2cdd98c2-c079-4880-9835-36d2c4a86de8\" (UID: \"2cdd98c2-c079-4880-9835-36d2c4a86de8\") " Oct 29 23:18:17.550577 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.550438 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-podinfo" pod "2cdd98c2-c079-4880-9835-36d2c4a86de8" (UID: "2cdd98c2-c079-4880-9835-36d2c4a86de8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:18:17.550577 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.550498 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-credential-socket" pod "be674d35-b876-4f7d-9ceb-8cd285a3f2db" (UID: "be674d35-b876-4f7d-9ceb-8cd285a3f2db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.551821 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.550979 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-credential-socket" pod "2cdd98c2-c079-4880-9835-36d2c4a86de8" (UID: "2cdd98c2-c079-4880-9835-36d2c4a86de8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.551821 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.551421 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-workload-certs" pod "2cdd98c2-c079-4880-9835-36d2c4a86de8" (UID: "2cdd98c2-c079-4880-9835-36d2c4a86de8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.551821 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.551612 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-data" pod "be674d35-b876-4f7d-9ceb-8cd285a3f2db" (UID: "be674d35-b876-4f7d-9ceb-8cd285a3f2db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.551821 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.551666 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cdd98c2-c079-4880-9835-36d2c4a86de8-istiod-ca-cert" pod "2cdd98c2-c079-4880-9835-36d2c4a86de8" (UID: "2cdd98c2-c079-4880-9835-36d2c4a86de8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:17.552158 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.551906 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-data" pod "2cdd98c2-c079-4880-9835-36d2c4a86de8" (UID: "2cdd98c2-c079-4880-9835-36d2c4a86de8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.552158 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.552071 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-workload-socket" pod "2cdd98c2-c079-4880-9835-36d2c4a86de8" (UID: "2cdd98c2-c079-4880-9835-36d2c4a86de8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.552304 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.552252 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-workload-certs" pod "be674d35-b876-4f7d-9ceb-8cd285a3f2db" (UID: "be674d35-b876-4f7d-9ceb-8cd285a3f2db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.552780 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.552492 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-workload-socket" pod "be674d35-b876-4f7d-9ceb-8cd285a3f2db" (UID: "be674d35-b876-4f7d-9ceb-8cd285a3f2db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.553499 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.553461 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istiod-ca-cert" pod "be674d35-b876-4f7d-9ceb-8cd285a3f2db" (UID: "be674d35-b876-4f7d-9ceb-8cd285a3f2db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:17.554051 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.554023 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-token" pod "be674d35-b876-4f7d-9ceb-8cd285a3f2db" (UID: "be674d35-b876-4f7d-9ceb-8cd285a3f2db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:17.557807 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.557760 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-token" pod "2cdd98c2-c079-4880-9835-36d2c4a86de8" (UID: "2cdd98c2-c079-4880-9835-36d2c4a86de8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:17.557807 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.557763 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-envoy" pod "2cdd98c2-c079-4880-9835-36d2c4a86de8" (UID: "2cdd98c2-c079-4880-9835-36d2c4a86de8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.557980 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.557901 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be674d35-b876-4f7d-9ceb-8cd285a3f2db-kube-api-access-nfvlk" pod "be674d35-b876-4f7d-9ceb-8cd285a3f2db" (UID: "be674d35-b876-4f7d-9ceb-8cd285a3f2db"). InnerVolumeSpecName "kube-api-access-nfvlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:17.558872 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.558850 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-podinfo" pod "be674d35-b876-4f7d-9ceb-8cd285a3f2db" (UID: "be674d35-b876-4f7d-9ceb-8cd285a3f2db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:18:17.560827 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.560806 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-envoy" pod "be674d35-b876-4f7d-9ceb-8cd285a3f2db" (UID: "be674d35-b876-4f7d-9ceb-8cd285a3f2db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:18:17.562540 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.562517 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cdd98c2-c079-4880-9835-36d2c4a86de8-kube-api-access-8gxdc" pod "2cdd98c2-c079-4880-9835-36d2c4a86de8" (UID: "2cdd98c2-c079-4880-9835-36d2c4a86de8"). InnerVolumeSpecName "kube-api-access-8gxdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:17.640925 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.640873 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.640925 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.640918 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gxdc\" (UniqueName: \"kubernetes.io/projected/2cdd98c2-c079-4880-9835-36d2c4a86de8-kube-api-access-8gxdc\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.640925 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.640935 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.641312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.640950 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.641312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.640964 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.641312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.640978 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.641312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.640991 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.641312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.641005 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.641312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.641017 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.641312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.641030 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cdd98c2-c079-4880-9835-36d2c4a86de8-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.641312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.641046 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.641312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.641059 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cdd98c2-c079-4880-9835-36d2c4a86de8-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.641312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.641072 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.641312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.641088 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfvlk\" (UniqueName: \"kubernetes.io/projected/be674d35-b876-4f7d-9ceb-8cd285a3f2db-kube-api-access-nfvlk\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.641312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.641101 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.641312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.641115 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be674d35-b876-4f7d-9ceb-8cd285a3f2db-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.641312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.641128 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be674d35-b876-4f7d-9ceb-8cd285a3f2db-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:18:17.641312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:17.641142 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cdd98c2-c079-4880-9835-36d2c4a86de8-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:18:18.250154 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:18.250070 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn" Oct 29 23:18:18.390843 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:18.390814 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" Oct 29 23:18:18.391268 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:18.390845 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd" event={"ID":"be674d35-b876-4f7d-9ceb-8cd285a3f2db","Type":"ContainerDied","Data":"4581c30751f7b7a4d1c855ea037472ba0a5591ca2d9242d8190c75bf96b645ef"} Oct 29 23:18:18.391268 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:18.390914 4821 scope.go:122] "RemoveContainer" containerID="c2003f19cf7d6a4f05f4fb76fb6a106b857e911e3f85ecc5d14a785250bcf519" Oct 29 23:18:18.391268 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:18.391085 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk" Oct 29 23:18:18.494941 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:18.494911 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd"] Oct 29 23:18:18.499622 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:18.499600 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-k7smd"] Oct 29 23:18:18.546745 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:18.546657 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk"] Oct 29 23:18:18.551011 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:18.550987 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-ngdqk"] Oct 29 23:18:20.172696 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:20.172658 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cdd98c2-c079-4880-9835-36d2c4a86de8" path="/var/lib/kubelet/pods/2cdd98c2-c079-4880-9835-36d2c4a86de8/volumes" Oct 29 23:18:20.173490 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:20.173469 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be674d35-b876-4f7d-9ceb-8cd285a3f2db" path="/var/lib/kubelet/pods/be674d35-b876-4f7d-9ceb-8cd285a3f2db/volumes" Oct 29 23:18:23.889409 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:18:23.889364 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:18:24.672785 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:24.671152 4821 patch_prober.go:29] interesting pod/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": read tcp 10.133.0.2:45344->10.133.0.235:15021: read: connection reset by peer" start-of-body= Oct 29 23:18:24.672785 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:24.671238 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" podUID="c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": read tcp 10.133.0.2:45344->10.133.0.235:15021: read: connection reset by peer" Oct 29 23:18:25.485023 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:25.483011 4821 generic.go:356] "Generic (PLEG): container finished" podID="c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" containerID="54363f00966d104348ac4181ee49432d4a9e9f1785ac761321df420926789766" exitCode=0 Oct 29 23:18:25.485023 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:25.483230 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" event={"ID":"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe","Type":"ContainerDied","Data":"54363f00966d104348ac4181ee49432d4a9e9f1785ac761321df420926789766"} Oct 29 23:18:25.485023 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:25.483867 4821 scope.go:122] "RemoveContainer" containerID="54363f00966d104348ac4181ee49432d4a9e9f1785ac761321df420926789766" Oct 29 23:18:25.487274 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:25.487130 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:18:25.487274 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:25.487253 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:18:25.487497 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:25.487309 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:18:26.493759 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:26.493721 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" event={"ID":"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe","Type":"ContainerStarted","Data":"c2422e9076a205fb6c3711163100b56c37c8faca0b2d7d48f4ac613e44633e8c"} Oct 29 23:18:27.156960 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:27.156893 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:18:27.157666 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:27.157635 4821 patch_prober.go:29] interesting pod/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 29 23:18:27.157912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:27.157885 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" podUID="c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 29 23:18:28.155434 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:28.155391 4821 patch_prober.go:29] interesting pod/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 29 23:18:28.155859 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:28.155449 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" podUID="c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 29 23:18:29.155737 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:29.155701 4821 patch_prober.go:29] interesting pod/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 29 23:18:29.156163 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:29.155759 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" podUID="c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 29 23:18:30.143118 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:30.143087 4821 scope.go:122] "RemoveContainer" containerID="c9aae7a6235e709152958c1867747e4fa2f852bd6d7f2a2dabfc43b67fa20e64" Oct 29 23:18:30.155301 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:30.155280 4821 scope.go:122] "RemoveContainer" containerID="c933808702e895530f39614d0d39c5921e1a1524d5be2eba7855bd85a087e097" Oct 29 23:18:30.155579 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:30.155552 4821 patch_prober.go:29] interesting pod/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 29 23:18:30.155684 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:30.155626 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" podUID="c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 29 23:18:31.155502 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:31.155461 4821 patch_prober.go:29] interesting pod/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 29 23:18:31.155952 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:31.155525 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" podUID="c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 29 23:18:32.172385 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:32.172347 4821 patch_prober.go:29] interesting pod/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body= Oct 29 23:18:32.173093 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:32.173057 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" podUID="c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" Oct 29 23:18:33.160169 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:33.160140 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:18:33.160663 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:33.160634 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:18:33.161372 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:33.161350 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:18:36.422294 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.422245 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k"] Oct 29 23:18:36.423417 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.423382 4821 cpu_manager.go:414] "containerMap: removing container" podUID="be674d35-b876-4f7d-9ceb-8cd285a3f2db" containerName="istio-proxy" Oct 29 23:18:36.423417 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.423416 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be674d35-b876-4f7d-9ceb-8cd285a3f2db" containerName="istio-proxy" Oct 29 23:18:36.423627 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.423534 4821 cpu_manager.go:414] "containerMap: removing container" podUID="2cdd98c2-c079-4880-9835-36d2c4a86de8" containerName="istio-proxy" Oct 29 23:18:36.423627 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.423544 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cdd98c2-c079-4880-9835-36d2c4a86de8" containerName="istio-proxy" Oct 29 23:18:36.423769 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.423736 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cdd98c2-c079-4880-9835-36d2c4a86de8" containerName="istio-proxy" Oct 29 23:18:36.423769 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.423756 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="be674d35-b876-4f7d-9ceb-8cd285a3f2db" containerName="istio-proxy" Oct 29 23:18:36.430394 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.430371 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:18:36.446716 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.446684 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k"] Oct 29 23:18:36.452821 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.452791 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-462fb2\"/\"test-model-462fb2-kserve-self-signed-certs\"" Oct 29 23:18:36.453033 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.452916 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-462fb2\"/\"default-dockercfg-7bfk7\"" Oct 29 23:18:36.460411 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.460364 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-462fb2\"/\"openshift-service-ca.crt\"" Oct 29 23:18:36.460601 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.460583 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-462fb2\"/\"kube-root-ca.crt\"" Oct 29 23:18:36.536349 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.536309 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-dshm\") pod \"test-model-462fb2-kserve-7b9c4d675-vn84k\" (UID: \"57f0a918-d7f5-4169-b2a9-106edf115d5e\") " pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:18:36.536548 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.536376 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-tmp-dir\") pod \"test-model-462fb2-kserve-7b9c4d675-vn84k\" (UID: \"57f0a918-d7f5-4169-b2a9-106edf115d5e\") " pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:18:36.536603 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.536590 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-home\") pod \"test-model-462fb2-kserve-7b9c4d675-vn84k\" (UID: \"57f0a918-d7f5-4169-b2a9-106edf115d5e\") " pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:18:36.536752 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.536733 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f0a918-d7f5-4169-b2a9-106edf115d5e-tls-certs\") pod \"test-model-462fb2-kserve-7b9c4d675-vn84k\" (UID: \"57f0a918-d7f5-4169-b2a9-106edf115d5e\") " pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:18:36.536794 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.536782 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpnv9\" (UniqueName: \"kubernetes.io/projected/57f0a918-d7f5-4169-b2a9-106edf115d5e-kube-api-access-bpnv9\") pod \"test-model-462fb2-kserve-7b9c4d675-vn84k\" (UID: \"57f0a918-d7f5-4169-b2a9-106edf115d5e\") " pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:18:36.537057 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.537032 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-model-cache\") pod \"test-model-462fb2-kserve-7b9c4d675-vn84k\" (UID: \"57f0a918-d7f5-4169-b2a9-106edf115d5e\") " pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:18:36.638427 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.638329 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-model-cache\") pod \"test-model-462fb2-kserve-7b9c4d675-vn84k\" (UID: \"57f0a918-d7f5-4169-b2a9-106edf115d5e\") " pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:18:36.638427 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.638395 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-dshm\") pod \"test-model-462fb2-kserve-7b9c4d675-vn84k\" (UID: \"57f0a918-d7f5-4169-b2a9-106edf115d5e\") " pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:18:36.638427 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.638427 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-tmp-dir\") pod \"test-model-462fb2-kserve-7b9c4d675-vn84k\" (UID: \"57f0a918-d7f5-4169-b2a9-106edf115d5e\") " pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:18:36.638758 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.638542 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-home\") pod \"test-model-462fb2-kserve-7b9c4d675-vn84k\" (UID: \"57f0a918-d7f5-4169-b2a9-106edf115d5e\") " pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:18:36.638758 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.638646 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f0a918-d7f5-4169-b2a9-106edf115d5e-tls-certs\") pod \"test-model-462fb2-kserve-7b9c4d675-vn84k\" (UID: \"57f0a918-d7f5-4169-b2a9-106edf115d5e\") " pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:18:36.638758 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.638687 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpnv9\" (UniqueName: \"kubernetes.io/projected/57f0a918-d7f5-4169-b2a9-106edf115d5e-kube-api-access-bpnv9\") pod \"test-model-462fb2-kserve-7b9c4d675-vn84k\" (UID: \"57f0a918-d7f5-4169-b2a9-106edf115d5e\") " pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:18:36.638928 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.638824 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-model-cache\") pod \"test-model-462fb2-kserve-7b9c4d675-vn84k\" (UID: \"57f0a918-d7f5-4169-b2a9-106edf115d5e\") " pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:18:36.638995 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.638926 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-tmp-dir\") pod \"test-model-462fb2-kserve-7b9c4d675-vn84k\" (UID: \"57f0a918-d7f5-4169-b2a9-106edf115d5e\") " pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:18:36.639364 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.639338 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-home\") pod \"test-model-462fb2-kserve-7b9c4d675-vn84k\" (UID: \"57f0a918-d7f5-4169-b2a9-106edf115d5e\") " pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:18:36.659242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.657593 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-dshm\") pod \"test-model-462fb2-kserve-7b9c4d675-vn84k\" (UID: \"57f0a918-d7f5-4169-b2a9-106edf115d5e\") " pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:18:36.659242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.658247 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f0a918-d7f5-4169-b2a9-106edf115d5e-tls-certs\") pod \"test-model-462fb2-kserve-7b9c4d675-vn84k\" (UID: \"57f0a918-d7f5-4169-b2a9-106edf115d5e\") " pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:18:36.694149 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.684960 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpnv9\" (UniqueName: \"kubernetes.io/projected/57f0a918-d7f5-4169-b2a9-106edf115d5e-kube-api-access-bpnv9\") pod \"test-model-462fb2-kserve-7b9c4d675-vn84k\" (UID: \"57f0a918-d7f5-4169-b2a9-106edf115d5e\") " pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:18:36.747611 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:36.747461 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:18:37.236937 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:37.236909 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k"] Oct 29 23:18:37.237426 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:18:37.237399 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57f0a918_d7f5_4169_b2a9_106edf115d5e.slice/crio-e3538bc95cf89d01b161c5f426c81f8b250ab3f6045a2261b0d05e7445570bc2 WatchSource:0}: Error finding container e3538bc95cf89d01b161c5f426c81f8b250ab3f6045a2261b0d05e7445570bc2: Status 404 returned error can't find the container with id e3538bc95cf89d01b161c5f426c81f8b250ab3f6045a2261b0d05e7445570bc2 Oct 29 23:18:37.704267 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:37.697452 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" event={"ID":"57f0a918-d7f5-4169-b2a9-106edf115d5e","Type":"ContainerStarted","Data":"1e579c6a4836ff346b5445091f7302701a5a73673e436fe3a3ca85475d46662b"} Oct 29 23:18:37.704267 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:37.697587 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" event={"ID":"57f0a918-d7f5-4169-b2a9-106edf115d5e","Type":"ContainerStarted","Data":"e3538bc95cf89d01b161c5f426c81f8b250ab3f6045a2261b0d05e7445570bc2"} Oct 29 23:18:37.753225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:37.751354 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:18:44.163153 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:44.163071 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" podStartSLOduration=8.163034522 podStartE2EDuration="8.163034522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:18:37.747867126 +0000 UTC m=+2171.688415037" watchObservedRunningTime="2026-10-29 23:18:44.163034522 +0000 UTC m=+2178.103582433" Oct 29 23:18:44.172729 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:44.172700 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf"] Oct 29 23:18:44.177529 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:44.177508 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" Oct 29 23:18:44.207063 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:44.205700 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58s9m\" (UniqueName: \"kubernetes.io/projected/0fd71b7c-e947-4db2-97fb-e16431ec4a37-kube-api-access-58s9m\") pod \"maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf\" (UID: \"0fd71b7c-e947-4db2-97fb-e16431ec4a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" Oct 29 23:18:44.207063 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:44.205850 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf"] Oct 29 23:18:44.207063 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:44.205960 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fd71b7c-e947-4db2-97fb-e16431ec4a37-maas-api-tls\") pod \"maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf\" (UID: \"0fd71b7c-e947-4db2-97fb-e16431ec4a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" Oct 29 23:18:44.207063 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:44.205997 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fd71b7c-e947-4db2-97fb-e16431ec4a37-metrics-certs\") pod \"maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf\" (UID: \"0fd71b7c-e947-4db2-97fb-e16431ec4a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" Oct 29 23:18:44.207063 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:44.206094 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd71b7c-e947-4db2-97fb-e16431ec4a37-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf\" (UID: \"0fd71b7c-e947-4db2-97fb-e16431ec4a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" Oct 29 23:18:44.209902 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:44.209877 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a9ae14-serving-cert\"" Oct 29 23:18:44.307695 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:44.307663 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fd71b7c-e947-4db2-97fb-e16431ec4a37-maas-api-tls\") pod \"maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf\" (UID: \"0fd71b7c-e947-4db2-97fb-e16431ec4a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" Oct 29 23:18:44.307917 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:44.307767 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fd71b7c-e947-4db2-97fb-e16431ec4a37-metrics-certs\") pod \"maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf\" (UID: \"0fd71b7c-e947-4db2-97fb-e16431ec4a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" Oct 29 23:18:44.307917 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:44.307870 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd71b7c-e947-4db2-97fb-e16431ec4a37-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf\" (UID: \"0fd71b7c-e947-4db2-97fb-e16431ec4a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" Oct 29 23:18:44.308061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:44.308023 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58s9m\" (UniqueName: \"kubernetes.io/projected/0fd71b7c-e947-4db2-97fb-e16431ec4a37-kube-api-access-58s9m\") pod \"maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf\" (UID: \"0fd71b7c-e947-4db2-97fb-e16431ec4a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" Oct 29 23:18:44.308609 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:44.308586 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd71b7c-e947-4db2-97fb-e16431ec4a37-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf\" (UID: \"0fd71b7c-e947-4db2-97fb-e16431ec4a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" Oct 29 23:18:44.310350 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:44.310318 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fd71b7c-e947-4db2-97fb-e16431ec4a37-metrics-certs\") pod \"maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf\" (UID: \"0fd71b7c-e947-4db2-97fb-e16431ec4a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" Oct 29 23:18:44.310479 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:44.310362 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fd71b7c-e947-4db2-97fb-e16431ec4a37-maas-api-tls\") pod \"maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf\" (UID: \"0fd71b7c-e947-4db2-97fb-e16431ec4a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" Oct 29 23:18:44.332631 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:44.332601 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58s9m\" (UniqueName: \"kubernetes.io/projected/0fd71b7c-e947-4db2-97fb-e16431ec4a37-kube-api-access-58s9m\") pod \"maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf\" (UID: \"0fd71b7c-e947-4db2-97fb-e16431ec4a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" Oct 29 23:18:44.516186 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:44.516080 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" Oct 29 23:18:44.842101 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:44.839674 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf"] Oct 29 23:18:44.842653 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:18:44.842595 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd71b7c_e947_4db2_97fb_e16431ec4a37.slice/crio-f15a7bbd70ec8d1fe2a6dc2604a41df166465e545d60beaac859c8eb40d45829 WatchSource:0}: Error finding container f15a7bbd70ec8d1fe2a6dc2604a41df166465e545d60beaac859c8eb40d45829: Status 404 returned error can't find the container with id f15a7bbd70ec8d1fe2a6dc2604a41df166465e545d60beaac859c8eb40d45829 Oct 29 23:18:45.766457 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:45.766414 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" event={"ID":"0fd71b7c-e947-4db2-97fb-e16431ec4a37","Type":"ContainerStarted","Data":"2a5e11fbdef469caa659a31d5ca210a0f06a7256cf679ac7afbdde0cf83883a3"} Oct 29 23:18:45.766896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:45.766489 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" event={"ID":"0fd71b7c-e947-4db2-97fb-e16431ec4a37","Type":"ContainerStarted","Data":"f15a7bbd70ec8d1fe2a6dc2604a41df166465e545d60beaac859c8eb40d45829"} Oct 29 23:18:45.766896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:45.766519 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" Oct 29 23:18:45.814991 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:45.814935 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" podStartSLOduration=1.423739696 podStartE2EDuration="1.814918575s" totalImagesPullingTime="391.178879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:18:45.811284535 +0000 UTC m=+2179.751832446" watchObservedRunningTime="2026-10-29 23:18:45.814918575 +0000 UTC m=+2179.755466486" Oct 29 23:18:46.748572 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:46.748541 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:18:46.761220 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:46.761180 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:18:46.786370 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:46.786331 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:18:50.718193 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:50.718155 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4"] Oct 29 23:18:50.722849 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:50.722825 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" Oct 29 23:18:50.752940 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:50.752911 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a37090-serving-cert\"" Oct 29 23:18:50.758777 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:50.758746 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4"] Oct 29 23:18:50.780078 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:50.780046 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49ae9910-d400-4cec-b171-d15541fa2789-maas-api-tls\") pod \"maas-api-e2e-inf-a-a37090-fb7866b54-l42t4\" (UID: \"49ae9910-d400-4cec-b171-d15541fa2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" Oct 29 23:18:50.780293 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:50.780169 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ae9910-d400-4cec-b171-d15541fa2789-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a37090-fb7866b54-l42t4\" (UID: \"49ae9910-d400-4cec-b171-d15541fa2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" Oct 29 23:18:50.780567 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:50.780537 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49ae9910-d400-4cec-b171-d15541fa2789-metrics-certs\") pod \"maas-api-e2e-inf-a-a37090-fb7866b54-l42t4\" (UID: \"49ae9910-d400-4cec-b171-d15541fa2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" Oct 29 23:18:50.780697 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:50.780632 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p5jp\" (UniqueName: \"kubernetes.io/projected/49ae9910-d400-4cec-b171-d15541fa2789-kube-api-access-2p5jp\") pod \"maas-api-e2e-inf-a-a37090-fb7866b54-l42t4\" (UID: \"49ae9910-d400-4cec-b171-d15541fa2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" Oct 29 23:18:50.882686 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:50.882649 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49ae9910-d400-4cec-b171-d15541fa2789-metrics-certs\") pod \"maas-api-e2e-inf-a-a37090-fb7866b54-l42t4\" (UID: \"49ae9910-d400-4cec-b171-d15541fa2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" Oct 29 23:18:50.882908 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:50.882729 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p5jp\" (UniqueName: \"kubernetes.io/projected/49ae9910-d400-4cec-b171-d15541fa2789-kube-api-access-2p5jp\") pod \"maas-api-e2e-inf-a-a37090-fb7866b54-l42t4\" (UID: \"49ae9910-d400-4cec-b171-d15541fa2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" Oct 29 23:18:50.882908 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:50.882852 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49ae9910-d400-4cec-b171-d15541fa2789-maas-api-tls\") pod \"maas-api-e2e-inf-a-a37090-fb7866b54-l42t4\" (UID: \"49ae9910-d400-4cec-b171-d15541fa2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" Oct 29 23:18:50.882908 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:50.882902 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ae9910-d400-4cec-b171-d15541fa2789-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a37090-fb7866b54-l42t4\" (UID: \"49ae9910-d400-4cec-b171-d15541fa2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" Oct 29 23:18:50.883658 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:50.883630 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ae9910-d400-4cec-b171-d15541fa2789-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a37090-fb7866b54-l42t4\" (UID: \"49ae9910-d400-4cec-b171-d15541fa2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" Oct 29 23:18:50.885462 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:50.885440 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49ae9910-d400-4cec-b171-d15541fa2789-metrics-certs\") pod \"maas-api-e2e-inf-a-a37090-fb7866b54-l42t4\" (UID: \"49ae9910-d400-4cec-b171-d15541fa2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" Oct 29 23:18:50.885697 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:50.885674 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49ae9910-d400-4cec-b171-d15541fa2789-maas-api-tls\") pod \"maas-api-e2e-inf-a-a37090-fb7866b54-l42t4\" (UID: \"49ae9910-d400-4cec-b171-d15541fa2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" Oct 29 23:18:50.908627 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:50.908595 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p5jp\" (UniqueName: \"kubernetes.io/projected/49ae9910-d400-4cec-b171-d15541fa2789-kube-api-access-2p5jp\") pod \"maas-api-e2e-inf-a-a37090-fb7866b54-l42t4\" (UID: \"49ae9910-d400-4cec-b171-d15541fa2789\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" Oct 29 23:18:51.064270 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:51.049348 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" Oct 29 23:18:51.406973 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:51.406947 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4"] Oct 29 23:18:51.407449 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:18:51.407427 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ae9910_d400_4cec_b171_d15541fa2789.slice/crio-a678b9b58f0ea6ce66d1fb9be0836d9705886e0b9eff785dfcced87031151c4d WatchSource:0}: Error finding container a678b9b58f0ea6ce66d1fb9be0836d9705886e0b9eff785dfcced87031151c4d: Status 404 returned error can't find the container with id a678b9b58f0ea6ce66d1fb9be0836d9705886e0b9eff785dfcced87031151c4d Oct 29 23:18:51.781652 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:51.781627 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" Oct 29 23:18:51.824720 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:51.824677 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" event={"ID":"49ae9910-d400-4cec-b171-d15541fa2789","Type":"ContainerStarted","Data":"8cbc885b699a94e28f961b7c29687941b18824cf7cf5210919f4f11ce24ba465"} Oct 29 23:18:51.824924 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:51.824739 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" event={"ID":"49ae9910-d400-4cec-b171-d15541fa2789","Type":"ContainerStarted","Data":"a678b9b58f0ea6ce66d1fb9be0836d9705886e0b9eff785dfcced87031151c4d"} Oct 29 23:18:51.825018 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:51.824916 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" Oct 29 23:18:51.880341 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:51.880270 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" podStartSLOduration=1.5759699980000001 podStartE2EDuration="1.880191293s" totalImagesPullingTime="304.221295ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:18:51.872440176 +0000 UTC m=+2185.812988088" watchObservedRunningTime="2026-10-29 23:18:51.880191293 +0000 UTC m=+2185.820739205" Oct 29 23:18:53.888695 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:18:53.888659 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:18:57.841027 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:18:57.840990 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" Oct 29 23:19:14.721858 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:14.721492 4821 patch_prober.go:29] interesting pod/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": read tcp 10.133.0.2:56824->10.133.0.244:15021: read: connection reset by peer" start-of-body= Oct 29 23:19:14.721858 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:14.721555 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" podUID="cf2ae83e-0943-4642-aa5f-2378b18bfe9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": read tcp 10.133.0.2:56824->10.133.0.244:15021: read: connection reset by peer" Oct 29 23:19:15.143411 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:15.143369 4821 generic.go:356] "Generic (PLEG): container finished" podID="cf2ae83e-0943-4642-aa5f-2378b18bfe9d" containerID="f14eeccb9ce53195c36db0341645d92bdfbb84d24077d5e4f8723b67cc84a96d" exitCode=0 Oct 29 23:19:15.143608 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:15.143428 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" event={"ID":"cf2ae83e-0943-4642-aa5f-2378b18bfe9d","Type":"ContainerDied","Data":"f14eeccb9ce53195c36db0341645d92bdfbb84d24077d5e4f8723b67cc84a96d"} Oct 29 23:19:15.143891 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:15.143877 4821 scope.go:122] "RemoveContainer" containerID="f14eeccb9ce53195c36db0341645d92bdfbb84d24077d5e4f8723b67cc84a96d" Oct 29 23:19:15.146046 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:15.146017 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:19:15.146121 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:15.146093 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:19:15.146162 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:15.146128 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:19:16.196301 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:16.196261 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" event={"ID":"cf2ae83e-0943-4642-aa5f-2378b18bfe9d","Type":"ContainerStarted","Data":"4142e8fc2469283bcbc13d35d144251eafec43077d73b35b9a5c0120f9dadcd5"} Oct 29 23:19:16.797042 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:16.797003 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:19:16.797330 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:16.797306 4821 patch_prober.go:29] interesting pod/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body= Oct 29 23:19:16.797428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:16.797363 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" podUID="cf2ae83e-0943-4642-aa5f-2378b18bfe9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" Oct 29 23:19:17.797349 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:17.797315 4821 patch_prober.go:29] interesting pod/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body= Oct 29 23:19:17.797779 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:17.797380 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" podUID="cf2ae83e-0943-4642-aa5f-2378b18bfe9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" Oct 29 23:19:18.797233 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:18.797163 4821 patch_prober.go:29] interesting pod/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body= Oct 29 23:19:18.797431 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:18.797249 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" podUID="cf2ae83e-0943-4642-aa5f-2378b18bfe9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" Oct 29 23:19:19.797327 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:19.797249 4821 patch_prober.go:29] interesting pod/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body= Oct 29 23:19:19.797327 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:19.797311 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" podUID="cf2ae83e-0943-4642-aa5f-2378b18bfe9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" Oct 29 23:19:20.797819 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:20.797783 4821 patch_prober.go:29] interesting pod/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body= Oct 29 23:19:20.798323 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:20.797835 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" podUID="cf2ae83e-0943-4642-aa5f-2378b18bfe9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" Oct 29 23:19:21.802697 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:21.802655 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:19:21.803282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:21.803216 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:19:21.804152 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:21.804131 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:19:23.933808 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:19:23.933754 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:19:30.246628 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:30.246597 4821 scope.go:122] "RemoveContainer" containerID="9cce07ab2eb3fb071565ff3268fa6e8adb951ebee99c3a3ead636a415cf2676d" Oct 29 23:19:31.581911 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:31.581873 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc"] Oct 29 23:19:31.582486 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:31.582092 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" podUID="2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc" containerName="istio-proxy" containerID="cri-o://dd1c23af9e77cd826fcbaecb1bdbd8d9504539921fa31975905e5b086f14f4e8" gracePeriod=30 Oct 29 23:19:33.491984 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:33.491953 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g"] Oct 29 23:19:33.492509 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:33.492165 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" podUID="4df31e57-c10a-4d14-b275-67b4983210ac" containerName="istio-proxy" containerID="cri-o://9054de3ab5d9fc2b4b84224f9d56ecaed4483ced8ba6003bac968ae4f646bff4" gracePeriod=30 Oct 29 23:19:35.483138 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:35.483102 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ls7vc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:19:35.484338 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:35.483182 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" podUID="2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:19:36.879524 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.879500 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 23:19:36.962759 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.962722 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c"] Oct 29 23:19:36.963928 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.963904 4821 cpu_manager.go:414] "containerMap: removing container" podUID="2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc" containerName="istio-proxy" Oct 29 23:19:36.964115 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.964101 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc" containerName="istio-proxy" Oct 29 23:19:36.964468 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.964448 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc" containerName="istio-proxy" Oct 29 23:19:36.968549 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.968526 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c" Oct 29 23:19:36.985573 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.985548 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-dcacb9-serving-cert\"" Oct 29 23:19:36.989716 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.989686 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c"] Oct 29 23:19:36.992836 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.992810 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-token\" (UniqueName: \"kubernetes.io/projected/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-token\") pod \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " Oct 29 23:19:36.993001 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.992947 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-credential-socket\") pod \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " Oct 29 23:19:36.993091 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.993008 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istiod-ca-cert\") pod \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " Oct 29 23:19:36.993159 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.993143 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-data\") pod \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " Oct 29 23:19:36.993239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.993181 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-envoy\") pod \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " Oct 29 23:19:36.993304 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.993282 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-kube-api-access-d6dcs\" (UniqueName: \"kubernetes.io/projected/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-kube-api-access-d6dcs\") pod \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " Oct 29 23:19:36.993425 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.993407 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-podinfo\") pod \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " Oct 29 23:19:36.993513 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.993492 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-workload-socket\") pod \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " Oct 29 23:19:36.993584 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.993569 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-workload-certs\") pod \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\" (UID: \"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc\") " Oct 29 23:19:36.993727 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.993701 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istiod-ca-cert" pod "2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc" (UID: "2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:19:36.993979 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.993955 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-data" pod "2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc" (UID: "2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:36.994334 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.994310 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-workload-certs" pod "2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc" (UID: "2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:36.994807 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.994783 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:19:36.995187 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.995158 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:19:36.995373 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.995357 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:19:36.995500 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.993958 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-credential-socket" pod "2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc" (UID: "2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:36.995932 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.995911 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-workload-socket" pod "2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc" (UID: "2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:36.997079 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.997055 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-podinfo" pod "2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc" (UID: "2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:19:36.998724 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:36.998689 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-token" pod "2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc" (UID: "2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:37.007819 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.007791 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-envoy" pod "2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc" (UID: "2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:37.009396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.009370 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-kube-api-access-d6dcs" pod "2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc" (UID: "2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc"). InnerVolumeSpecName "kube-api-access-d6dcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:37.097011 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.096968 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78cb6576-8c0f-4a8b-9d05-72de5333c31d-maas-api-tls\") pod \"maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c\" (UID: \"78cb6576-8c0f-4a8b-9d05-72de5333c31d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c" Oct 29 23:19:37.097192 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.097110 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78cb6576-8c0f-4a8b-9d05-72de5333c31d-metrics-certs\") pod \"maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c\" (UID: \"78cb6576-8c0f-4a8b-9d05-72de5333c31d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c" Oct 29 23:19:37.097192 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.097220 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ktww\" (UniqueName: \"kubernetes.io/projected/78cb6576-8c0f-4a8b-9d05-72de5333c31d-kube-api-access-2ktww\") pod \"maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c\" (UID: \"78cb6576-8c0f-4a8b-9d05-72de5333c31d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c" Oct 29 23:19:37.097375 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.097349 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78cb6576-8c0f-4a8b-9d05-72de5333c31d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c\" (UID: \"78cb6576-8c0f-4a8b-9d05-72de5333c31d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c" Oct 29 23:19:37.097676 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.097653 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:19:37.097770 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.097682 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:19:37.097770 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.097697 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:19:37.097770 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.097710 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6dcs\" (UniqueName: \"kubernetes.io/projected/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-kube-api-access-d6dcs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:19:37.097770 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.097724 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:19:37.097770 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.097737 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:19:37.199694 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.199660 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78cb6576-8c0f-4a8b-9d05-72de5333c31d-maas-api-tls\") pod \"maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c\" (UID: \"78cb6576-8c0f-4a8b-9d05-72de5333c31d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c" Oct 29 23:19:37.199889 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.199718 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78cb6576-8c0f-4a8b-9d05-72de5333c31d-metrics-certs\") pod \"maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c\" (UID: \"78cb6576-8c0f-4a8b-9d05-72de5333c31d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c" Oct 29 23:19:37.199889 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.199748 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ktww\" (UniqueName: \"kubernetes.io/projected/78cb6576-8c0f-4a8b-9d05-72de5333c31d-kube-api-access-2ktww\") pod \"maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c\" (UID: \"78cb6576-8c0f-4a8b-9d05-72de5333c31d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c" Oct 29 23:19:37.199889 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.199800 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78cb6576-8c0f-4a8b-9d05-72de5333c31d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c\" (UID: \"78cb6576-8c0f-4a8b-9d05-72de5333c31d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c" Oct 29 23:19:37.200499 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.200408 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78cb6576-8c0f-4a8b-9d05-72de5333c31d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c\" (UID: \"78cb6576-8c0f-4a8b-9d05-72de5333c31d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c" Oct 29 23:19:37.202328 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.202304 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78cb6576-8c0f-4a8b-9d05-72de5333c31d-maas-api-tls\") pod \"maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c\" (UID: \"78cb6576-8c0f-4a8b-9d05-72de5333c31d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c" Oct 29 23:19:37.202415 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.202350 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78cb6576-8c0f-4a8b-9d05-72de5333c31d-metrics-certs\") pod \"maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c\" (UID: \"78cb6576-8c0f-4a8b-9d05-72de5333c31d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c" Oct 29 23:19:37.223968 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.223943 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ktww\" (UniqueName: \"kubernetes.io/projected/78cb6576-8c0f-4a8b-9d05-72de5333c31d-kube-api-access-2ktww\") pod \"maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c\" (UID: \"78cb6576-8c0f-4a8b-9d05-72de5333c31d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c" Oct 29 23:19:37.285655 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.285621 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c" Oct 29 23:19:37.439364 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.439326 4821 generic.go:356] "Generic (PLEG): container finished" podID="2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc" containerID="dd1c23af9e77cd826fcbaecb1bdbd8d9504539921fa31975905e5b086f14f4e8" exitCode=0 Oct 29 23:19:37.439548 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.439419 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" event={"ID":"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc","Type":"ContainerDied","Data":"dd1c23af9e77cd826fcbaecb1bdbd8d9504539921fa31975905e5b086f14f4e8"} Oct 29 23:19:37.439548 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.439472 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" event={"ID":"2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc","Type":"ContainerDied","Data":"71a9d5a4f4554c733fd4f09d222f48a912c86b233ade3e4527160b4776d404e5"} Oct 29 23:19:37.439548 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.439502 4821 scope.go:122] "RemoveContainer" containerID="dd1c23af9e77cd826fcbaecb1bdbd8d9504539921fa31975905e5b086f14f4e8" Oct 29 23:19:37.439712 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.439693 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc" Oct 29 23:19:37.455748 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.455724 4821 scope.go:122] "RemoveContainer" containerID="dd1c23af9e77cd826fcbaecb1bdbd8d9504539921fa31975905e5b086f14f4e8" Oct 29 23:19:37.457693 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:19:37.457645 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd1c23af9e77cd826fcbaecb1bdbd8d9504539921fa31975905e5b086f14f4e8\": container with ID starting with dd1c23af9e77cd826fcbaecb1bdbd8d9504539921fa31975905e5b086f14f4e8 not found: ID does not exist" containerID="dd1c23af9e77cd826fcbaecb1bdbd8d9504539921fa31975905e5b086f14f4e8" Oct 29 23:19:37.457693 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.457681 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd1c23af9e77cd826fcbaecb1bdbd8d9504539921fa31975905e5b086f14f4e8"} err="failed to get container status \"dd1c23af9e77cd826fcbaecb1bdbd8d9504539921fa31975905e5b086f14f4e8\": rpc error: code = NotFound desc = could not find container \"dd1c23af9e77cd826fcbaecb1bdbd8d9504539921fa31975905e5b086f14f4e8\": container with ID starting with dd1c23af9e77cd826fcbaecb1bdbd8d9504539921fa31975905e5b086f14f4e8 not found: ID does not exist" Oct 29 23:19:37.501166 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.501135 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc"] Oct 29 23:19:37.505887 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.505861 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ls7vc"] Oct 29 23:19:37.603950 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:37.603920 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c"] Oct 29 23:19:37.605444 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:19:37.605417 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78cb6576_8c0f_4a8b_9d05_72de5333c31d.slice/crio-237a0546b93cb8a395606561562db1106c5f8eb41e2eb4eba3bdebda0b7f2e01 WatchSource:0}: Error finding container 237a0546b93cb8a395606561562db1106c5f8eb41e2eb4eba3bdebda0b7f2e01: Status 404 returned error can't find the container with id 237a0546b93cb8a395606561562db1106c5f8eb41e2eb4eba3bdebda0b7f2e01 Oct 29 23:19:38.175667 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:38.175590 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc" path="/var/lib/kubelet/pods/2afb440c-9fff-4ea8-8c9e-f9a3bddc2ffc/volumes" Oct 29 23:19:38.450353 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:38.450262 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c" event={"ID":"78cb6576-8c0f-4a8b-9d05-72de5333c31d","Type":"ContainerStarted","Data":"2304ae03eb4b964b87a3bf119a55e83af5baa4547d6babbedc024a52a7e024a1"} Oct 29 23:19:38.450353 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:38.450328 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c" event={"ID":"78cb6576-8c0f-4a8b-9d05-72de5333c31d","Type":"ContainerStarted","Data":"237a0546b93cb8a395606561562db1106c5f8eb41e2eb4eba3bdebda0b7f2e01"} Oct 29 23:19:38.450600 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:38.450511 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c" Oct 29 23:19:38.502565 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:38.502507 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c" podStartSLOduration=43.217693424 podStartE2EDuration="43.502491475s" totalImagesPullingTime="284.798051ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:38.49549501 +0000 UTC m=+2232.436042914" watchObservedRunningTime="2026-10-29 23:19:38.502491475 +0000 UTC m=+2232.443039424" Oct 29 23:19:38.856636 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:38.856607 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:19:38.923600 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:38.923567 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8"] Oct 29 23:19:38.924781 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:38.924754 4821 cpu_manager.go:414] "containerMap: removing container" podUID="4df31e57-c10a-4d14-b275-67b4983210ac" containerName="istio-proxy" Oct 29 23:19:38.924781 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:38.924783 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4df31e57-c10a-4d14-b275-67b4983210ac" containerName="istio-proxy" Oct 29 23:19:38.925044 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:38.925027 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="4df31e57-c10a-4d14-b275-67b4983210ac" containerName="istio-proxy" Oct 29 23:19:38.929475 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:38.929454 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8" Oct 29 23:19:38.949839 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:38.946091 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1a8107-serving-cert\"" Oct 29 23:19:38.951249 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:38.951219 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8"] Oct 29 23:19:39.040397 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.040307 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-workload-socket\") pod \"4df31e57-c10a-4d14-b275-67b4983210ac\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " Oct 29 23:19:39.040587 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.040395 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4df31e57-c10a-4d14-b275-67b4983210ac-istio-token\" (UniqueName: \"kubernetes.io/projected/4df31e57-c10a-4d14-b275-67b4983210ac-istio-token\") pod \"4df31e57-c10a-4d14-b275-67b4983210ac\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " Oct 29 23:19:39.040587 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.040515 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4df31e57-c10a-4d14-b275-67b4983210ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df31e57-c10a-4d14-b275-67b4983210ac-istio-podinfo\") pod \"4df31e57-c10a-4d14-b275-67b4983210ac\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " Oct 29 23:19:39.040735 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.040650 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-istio-envoy\") pod \"4df31e57-c10a-4d14-b275-67b4983210ac\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " Oct 29 23:19:39.040807 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.040751 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-credential-socket\") pod \"4df31e57-c10a-4d14-b275-67b4983210ac\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " Oct 29 23:19:39.040807 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.040784 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4df31e57-c10a-4d14-b275-67b4983210ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df31e57-c10a-4d14-b275-67b4983210ac-istiod-ca-cert\") pod \"4df31e57-c10a-4d14-b275-67b4983210ac\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " Oct 29 23:19:39.040926 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.040899 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4df31e57-c10a-4d14-b275-67b4983210ac-kube-api-access-wdj27\" (UniqueName: \"kubernetes.io/projected/4df31e57-c10a-4d14-b275-67b4983210ac-kube-api-access-wdj27\") pod \"4df31e57-c10a-4d14-b275-67b4983210ac\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " Oct 29 23:19:39.041045 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.041025 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-istio-data\") pod \"4df31e57-c10a-4d14-b275-67b4983210ac\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " Oct 29 23:19:39.041125 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.041110 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-workload-certs\") pod \"4df31e57-c10a-4d14-b275-67b4983210ac\" (UID: \"4df31e57-c10a-4d14-b275-67b4983210ac\") " Oct 29 23:19:39.041414 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.041389 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8\" (UID: \"1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8" Oct 29 23:19:39.041521 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.041482 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-maas-api-tls\") pod \"maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8\" (UID: \"1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8" Oct 29 23:19:39.041593 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.041544 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-workload-socket" pod "4df31e57-c10a-4d14-b275-67b4983210ac" (UID: "4df31e57-c10a-4d14-b275-67b4983210ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:39.041683 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.041657 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d4jm\" (UniqueName: \"kubernetes.io/projected/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-kube-api-access-9d4jm\") pod \"maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8\" (UID: \"1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8" Oct 29 23:19:39.041845 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.041817 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-credential-socket" pod "4df31e57-c10a-4d14-b275-67b4983210ac" (UID: "4df31e57-c10a-4d14-b275-67b4983210ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:39.042052 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.042028 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-metrics-certs\") pod \"maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8\" (UID: \"1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8" Oct 29 23:19:39.042159 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.042119 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df31e57-c10a-4d14-b275-67b4983210ac-istiod-ca-cert" pod "4df31e57-c10a-4d14-b275-67b4983210ac" (UID: "4df31e57-c10a-4d14-b275-67b4983210ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:19:39.042355 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.042333 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:19:39.042484 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.042361 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df31e57-c10a-4d14-b275-67b4983210ac-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:19:39.042484 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.042376 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:19:39.042702 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.042679 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-workload-certs" pod "4df31e57-c10a-4d14-b275-67b4983210ac" (UID: "4df31e57-c10a-4d14-b275-67b4983210ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:39.042892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.042870 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-istio-data" pod "4df31e57-c10a-4d14-b275-67b4983210ac" (UID: "4df31e57-c10a-4d14-b275-67b4983210ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:39.055032 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.055010 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4df31e57-c10a-4d14-b275-67b4983210ac-istio-podinfo" pod "4df31e57-c10a-4d14-b275-67b4983210ac" (UID: "4df31e57-c10a-4d14-b275-67b4983210ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:19:39.055155 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.055132 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-istio-envoy" pod "4df31e57-c10a-4d14-b275-67b4983210ac" (UID: "4df31e57-c10a-4d14-b275-67b4983210ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:19:39.055237 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.055166 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df31e57-c10a-4d14-b275-67b4983210ac-istio-token" pod "4df31e57-c10a-4d14-b275-67b4983210ac" (UID: "4df31e57-c10a-4d14-b275-67b4983210ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:39.056893 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.056830 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df31e57-c10a-4d14-b275-67b4983210ac-kube-api-access-wdj27" pod "4df31e57-c10a-4d14-b275-67b4983210ac" (UID: "4df31e57-c10a-4d14-b275-67b4983210ac"). InnerVolumeSpecName "kube-api-access-wdj27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:19:39.144353 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.144317 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-metrics-certs\") pod \"maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8\" (UID: \"1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8" Oct 29 23:19:39.144512 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.144449 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8\" (UID: \"1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8" Oct 29 23:19:39.144512 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.144490 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-maas-api-tls\") pod \"maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8\" (UID: \"1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8" Oct 29 23:19:39.144598 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.144549 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d4jm\" (UniqueName: \"kubernetes.io/projected/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-kube-api-access-9d4jm\") pod \"maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8\" (UID: \"1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8" Oct 29 23:19:39.144725 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.144710 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df31e57-c10a-4d14-b275-67b4983210ac-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:19:39.144779 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.144731 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:19:39.144779 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.144747 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdj27\" (UniqueName: \"kubernetes.io/projected/4df31e57-c10a-4d14-b275-67b4983210ac-kube-api-access-wdj27\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:19:39.144779 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.144759 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:19:39.144779 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.144774 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df31e57-c10a-4d14-b275-67b4983210ac-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:19:39.144976 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.144785 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df31e57-c10a-4d14-b275-67b4983210ac-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:19:39.145195 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.145125 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8\" (UID: \"1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8" Oct 29 23:19:39.147129 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.147106 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-maas-api-tls\") pod \"maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8\" (UID: \"1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8" Oct 29 23:19:39.147255 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.147144 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-metrics-certs\") pod \"maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8\" (UID: \"1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8" Oct 29 23:19:39.169341 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.169318 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d4jm\" (UniqueName: \"kubernetes.io/projected/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-kube-api-access-9d4jm\") pod \"maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8\" (UID: \"1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8" Oct 29 23:19:39.252742 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.252704 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8" Oct 29 23:19:39.468355 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.468323 4821 generic.go:356] "Generic (PLEG): container finished" podID="4df31e57-c10a-4d14-b275-67b4983210ac" containerID="9054de3ab5d9fc2b4b84224f9d56ecaed4483ced8ba6003bac968ae4f646bff4" exitCode=0 Oct 29 23:19:39.468559 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.468394 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" Oct 29 23:19:39.468559 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.468406 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" event={"ID":"4df31e57-c10a-4d14-b275-67b4983210ac","Type":"ContainerDied","Data":"9054de3ab5d9fc2b4b84224f9d56ecaed4483ced8ba6003bac968ae4f646bff4"} Oct 29 23:19:39.468559 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.468483 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g" event={"ID":"4df31e57-c10a-4d14-b275-67b4983210ac","Type":"ContainerDied","Data":"b5d0b59f2d66ae8c5455ac61b3953247ab9f110afdbbce3f9578dfbe846cf7b2"} Oct 29 23:19:39.468559 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.468506 4821 scope.go:122] "RemoveContainer" containerID="9054de3ab5d9fc2b4b84224f9d56ecaed4483ced8ba6003bac968ae4f646bff4" Oct 29 23:19:39.480878 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.480861 4821 scope.go:122] "RemoveContainer" containerID="9054de3ab5d9fc2b4b84224f9d56ecaed4483ced8ba6003bac968ae4f646bff4" Oct 29 23:19:39.481149 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:19:39.481124 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9054de3ab5d9fc2b4b84224f9d56ecaed4483ced8ba6003bac968ae4f646bff4\": container with ID starting with 9054de3ab5d9fc2b4b84224f9d56ecaed4483ced8ba6003bac968ae4f646bff4 not found: ID does not exist" containerID="9054de3ab5d9fc2b4b84224f9d56ecaed4483ced8ba6003bac968ae4f646bff4" Oct 29 23:19:39.481235 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.481157 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9054de3ab5d9fc2b4b84224f9d56ecaed4483ced8ba6003bac968ae4f646bff4"} err="failed to get container status \"9054de3ab5d9fc2b4b84224f9d56ecaed4483ced8ba6003bac968ae4f646bff4\": rpc error: code = NotFound desc = could not find container \"9054de3ab5d9fc2b4b84224f9d56ecaed4483ced8ba6003bac968ae4f646bff4\": container with ID starting with 9054de3ab5d9fc2b4b84224f9d56ecaed4483ced8ba6003bac968ae4f646bff4 not found: ID does not exist" Oct 29 23:19:39.527105 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.527078 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g"] Oct 29 23:19:39.539877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.539851 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jg75g"] Oct 29 23:19:39.582901 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:39.582873 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8"] Oct 29 23:19:39.583038 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:19:39.583002 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b2afc10_56bd_4ec9_bbbe_1ebd16f7d9f0.slice/crio-7b954c702eae4c07c2ebe43b5ca3c2fb4ab2cf710f99a3ef22a222abaaf3b51b WatchSource:0}: Error finding container 7b954c702eae4c07c2ebe43b5ca3c2fb4ab2cf710f99a3ef22a222abaaf3b51b: Status 404 returned error can't find the container with id 7b954c702eae4c07c2ebe43b5ca3c2fb4ab2cf710f99a3ef22a222abaaf3b51b Oct 29 23:19:40.174058 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:40.174023 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4df31e57-c10a-4d14-b275-67b4983210ac" path="/var/lib/kubelet/pods/4df31e57-c10a-4d14-b275-67b4983210ac/volumes" Oct 29 23:19:40.482587 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:40.482549 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8" event={"ID":"1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0","Type":"ContainerStarted","Data":"210cba46e83eb16c3f1e7861b0999c98d2b4141f505d681e297920a19c78e4c3"} Oct 29 23:19:40.483104 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:40.482618 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8" event={"ID":"1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0","Type":"ContainerStarted","Data":"7b954c702eae4c07c2ebe43b5ca3c2fb4ab2cf710f99a3ef22a222abaaf3b51b"} Oct 29 23:19:40.483104 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:40.482648 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8" Oct 29 23:19:40.529585 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:40.529535 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8" podStartSLOduration=35.145864107 podStartE2EDuration="35.529520361s" totalImagesPullingTime="383.656254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:40.527085665 +0000 UTC m=+2234.467633574" watchObservedRunningTime="2026-10-29 23:19:40.529520361 +0000 UTC m=+2234.470068271" Oct 29 23:19:44.474683 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:44.474653 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c" Oct 29 23:19:46.520124 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:46.520095 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8" Oct 29 23:19:53.888887 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:19:53.888848 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:19:57.044712 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:57.042900 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43add4fd-dnk8c"] Oct 29 23:19:57.077222 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:57.074656 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43add4fd-dnk8c"] Oct 29 23:19:58.184827 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:19:58.184791 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82cd32c0-9b49-4b74-8d05-d64ee8576c8d" path="/var/lib/kubelet/pods/82cd32c0-9b49-4b74-8d05-d64ee8576c8d/volumes" Oct 29 23:20:03.065423 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:03.065391 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46d37e92-phvgz"] Oct 29 23:20:03.072279 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:03.071675 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46d37e92-phvgz"] Oct 29 23:20:04.213240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:04.197158 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def6461d-2480-45dc-b7cd-9f7047855ca7" path="/var/lib/kubelet/pods/def6461d-2480-45dc-b7cd-9f7047855ca7/volumes" Oct 29 23:20:10.990334 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:10.989844 4821 patch_prober.go:29] interesting pod/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": read tcp 10.133.0.2:58664->10.133.0.252:15021: read: connection reset by peer" start-of-body= Oct 29 23:20:10.990334 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:10.989935 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" podUID="d560c3fe-b142-4ae7-a7da-c86260287c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": read tcp 10.133.0.2:58664->10.133.0.252:15021: read: connection reset by peer" Oct 29 23:20:11.041751 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:11.041714 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5783e09-9vm4d"] Oct 29 23:20:11.047655 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:11.047623 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5783e09-9vm4d"] Oct 29 23:20:11.197213 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:11.197168 4821 generic.go:356] "Generic (PLEG): container finished" podID="d560c3fe-b142-4ae7-a7da-c86260287c18" containerID="9c9a20c330af433a30b361d363749ffd131ec1fd73f3cd04bfe058a5c83d6199" exitCode=0 Oct 29 23:20:11.197419 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:11.197273 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" event={"ID":"d560c3fe-b142-4ae7-a7da-c86260287c18","Type":"ContainerDied","Data":"9c9a20c330af433a30b361d363749ffd131ec1fd73f3cd04bfe058a5c83d6199"} Oct 29 23:20:11.197829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:11.197810 4821 scope.go:122] "RemoveContainer" containerID="9c9a20c330af433a30b361d363749ffd131ec1fd73f3cd04bfe058a5c83d6199" Oct 29 23:20:11.200246 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:11.200185 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:20:11.200349 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:11.200283 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:20:11.200349 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:11.200319 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:20:11.943971 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:11.943938 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q"] Oct 29 23:20:11.944251 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:11.944184 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" podUID="4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad" containerName="istio-proxy" containerID="cri-o://6c6436c8e2d277135120be699fe00780fc1a1f8d03ec0dd25614743cc6f73893" gracePeriod=30 Oct 29 23:20:12.227280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:12.225942 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74bc6f99-ab50-4bd6-9b61-a8727d36f879" path="/var/lib/kubelet/pods/74bc6f99-ab50-4bd6-9b61-a8727d36f879/volumes" Oct 29 23:20:12.293941 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:12.293875 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" event={"ID":"d560c3fe-b142-4ae7-a7da-c86260287c18","Type":"ContainerStarted","Data":"ff13156500ef13b127a38a578d875fa9b1c88774d6b281bd03f7939afc9be6f6"} Oct 29 23:20:12.938225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:12.937298 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:20:12.939340 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:12.939307 4821 patch_prober.go:29] interesting pod/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body= Oct 29 23:20:12.939503 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:12.939385 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" podUID="d560c3fe-b142-4ae7-a7da-c86260287c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" Oct 29 23:20:13.935226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:13.934627 4821 patch_prober.go:29] interesting pod/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body= Oct 29 23:20:13.935226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:13.934691 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" podUID="d560c3fe-b142-4ae7-a7da-c86260287c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" Oct 29 23:20:14.940638 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:14.940596 4821 patch_prober.go:29] interesting pod/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body= Oct 29 23:20:14.941268 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:14.940667 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" podUID="d560c3fe-b142-4ae7-a7da-c86260287c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" Oct 29 23:20:15.934226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:15.933559 4821 patch_prober.go:29] interesting pod/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body= Oct 29 23:20:15.934226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:15.933675 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" podUID="d560c3fe-b142-4ae7-a7da-c86260287c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" Oct 29 23:20:16.936230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:16.932561 4821 patch_prober.go:29] interesting pod/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body= Oct 29 23:20:16.936230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:16.932630 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" podUID="d560c3fe-b142-4ae7-a7da-c86260287c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" Oct 29 23:20:17.279440 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.279412 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:20:17.387753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.387715 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-kube-api-access-br6x6\" (UniqueName: \"kubernetes.io/projected/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-kube-api-access-br6x6\") pod \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " Oct 29 23:20:17.387962 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.387793 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-credential-socket\") pod \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " Oct 29 23:20:17.387962 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.387946 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-workload-socket\") pod \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " Oct 29 23:20:17.388086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.388056 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-workload-certs\") pod \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " Oct 29 23:20:17.388086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.388066 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-credential-socket" pod "4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad" (UID: "4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:20:17.388215 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.388111 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istiod-ca-cert\") pod \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " Oct 29 23:20:17.388215 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.388141 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-podinfo\") pod \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " Oct 29 23:20:17.388352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.388243 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-workload-socket" pod "4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad" (UID: "4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:20:17.388352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.388278 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-token\" (UniqueName: \"kubernetes.io/projected/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-token\") pod \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " Oct 29 23:20:17.388352 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.388329 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-workload-certs" pod "4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad" (UID: "4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:20:17.388520 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.388379 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-envoy\") pod \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " Oct 29 23:20:17.388520 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.388430 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-data\") pod \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\" (UID: \"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad\") " Oct 29 23:20:17.388649 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.388511 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istiod-ca-cert" pod "4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad" (UID: "4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:20:17.389629 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.388867 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-data" pod "4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad" (UID: "4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:20:17.389629 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.389401 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:20:17.389629 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.389424 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:20:17.389629 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.389437 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:20:17.389629 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.389450 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:20:17.389629 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.389464 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:20:17.390920 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.390898 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-podinfo" pod "4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad" (UID: "4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:20:17.391073 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.391056 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-envoy" pod "4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad" (UID: "4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:20:17.391144 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.391129 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-kube-api-access-br6x6" pod "4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad" (UID: "4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad"). InnerVolumeSpecName "kube-api-access-br6x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:20:17.391772 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.391750 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-token" pod "4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad" (UID: "4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:20:17.407935 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.407907 4821 generic.go:356] "Generic (PLEG): container finished" podID="4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad" containerID="6c6436c8e2d277135120be699fe00780fc1a1f8d03ec0dd25614743cc6f73893" exitCode=0 Oct 29 23:20:17.408070 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.407964 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" Oct 29 23:20:17.408070 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.407989 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" event={"ID":"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad","Type":"ContainerDied","Data":"6c6436c8e2d277135120be699fe00780fc1a1f8d03ec0dd25614743cc6f73893"} Oct 29 23:20:17.408070 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.408045 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q" event={"ID":"4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad","Type":"ContainerDied","Data":"a8ba125434395b671461b77b8cac1517c17fcf265b67104e3f9e4fe695a08fcc"} Oct 29 23:20:17.408070 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.408068 4821 scope.go:122] "RemoveContainer" containerID="6c6436c8e2d277135120be699fe00780fc1a1f8d03ec0dd25614743cc6f73893" Oct 29 23:20:17.457226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.457182 4821 scope.go:122] "RemoveContainer" containerID="6c6436c8e2d277135120be699fe00780fc1a1f8d03ec0dd25614743cc6f73893" Oct 29 23:20:17.457550 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:20:17.457529 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6436c8e2d277135120be699fe00780fc1a1f8d03ec0dd25614743cc6f73893\": container with ID starting with 6c6436c8e2d277135120be699fe00780fc1a1f8d03ec0dd25614743cc6f73893 not found: ID does not exist" containerID="6c6436c8e2d277135120be699fe00780fc1a1f8d03ec0dd25614743cc6f73893" Oct 29 23:20:17.457630 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.457564 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6436c8e2d277135120be699fe00780fc1a1f8d03ec0dd25614743cc6f73893"} err="failed to get container status \"6c6436c8e2d277135120be699fe00780fc1a1f8d03ec0dd25614743cc6f73893\": rpc error: code = NotFound desc = could not find container \"6c6436c8e2d277135120be699fe00780fc1a1f8d03ec0dd25614743cc6f73893\": container with ID starting with 6c6436c8e2d277135120be699fe00780fc1a1f8d03ec0dd25614743cc6f73893 not found: ID does not exist" Oct 29 23:20:17.487476 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.487441 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q"] Oct 29 23:20:17.491391 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.491368 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:20:17.491391 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.491389 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:20:17.491609 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.491406 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:20:17.491609 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.491417 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br6x6\" (UniqueName: \"kubernetes.io/projected/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad-kube-api-access-br6x6\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:20:17.492126 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.492102 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fba83e-openshift-default-7544fcb869-bwv6q"] Oct 29 23:20:17.960280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.960164 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:20:17.961252 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.961194 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:20:17.963070 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:17.963049 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:20:18.188396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:18.188362 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad" path="/var/lib/kubelet/pods/4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad/volumes" Oct 29 23:20:23.970131 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:20:23.970082 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:20:25.092926 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.092886 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj"] Oct 29 23:20:25.093995 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.093971 4821 cpu_manager.go:414] "containerMap: removing container" podUID="4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad" containerName="istio-proxy" Oct 29 23:20:25.093995 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.093992 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad" containerName="istio-proxy" Oct 29 23:20:25.094243 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.094121 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bcf37ba-4f91-43c4-ac2d-2cc310f9b7ad" containerName="istio-proxy" Oct 29 23:20:25.098032 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.098003 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.116595 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.116573 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1056cc-openshift-default-dockercfg-6p8ms\"" Oct 29 23:20:25.137160 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.137130 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj"] Oct 29 23:20:25.248409 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.248364 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrtbs\" (UniqueName: \"kubernetes.io/projected/b26786c8-6181-41f2-a978-49b2123a690c-kube-api-access-lrtbs\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.248622 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.248467 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b26786c8-6181-41f2-a978-49b2123a690c-istio-podinfo\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.248690 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.248650 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-istio-envoy\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.248892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.248869 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-workload-socket\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.248991 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.248975 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b26786c8-6181-41f2-a978-49b2123a690c-istiod-ca-cert\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.249137 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.249110 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-workload-certs\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.249193 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.249166 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b26786c8-6181-41f2-a978-49b2123a690c-istio-token\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.249318 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.249301 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-credential-socket\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.249431 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.249416 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-istio-data\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.350958 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.350858 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b26786c8-6181-41f2-a978-49b2123a690c-istiod-ca-cert\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.350958 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.350932 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-workload-certs\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.351258 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.350965 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b26786c8-6181-41f2-a978-49b2123a690c-istio-token\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.351258 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.351126 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-credential-socket\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.351258 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.351251 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-istio-data\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.351442 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.351302 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrtbs\" (UniqueName: \"kubernetes.io/projected/b26786c8-6181-41f2-a978-49b2123a690c-kube-api-access-lrtbs\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.351442 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.351332 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b26786c8-6181-41f2-a978-49b2123a690c-istio-podinfo\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.351442 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.351381 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-istio-envoy\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.351611 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.351432 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-workload-certs\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.351611 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.351475 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-workload-socket\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.351876 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.351843 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b26786c8-6181-41f2-a978-49b2123a690c-istiod-ca-cert\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.352015 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.351882 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-workload-socket\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.352015 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.351894 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-credential-socket\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.352015 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.351961 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-istio-data\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.353910 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.353889 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-istio-envoy\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.354152 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.354131 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b26786c8-6181-41f2-a978-49b2123a690c-istio-podinfo\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.376134 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.376102 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrtbs\" (UniqueName: \"kubernetes.io/projected/b26786c8-6181-41f2-a978-49b2123a690c-kube-api-access-lrtbs\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.376294 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.376174 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b26786c8-6181-41f2-a978-49b2123a690c-istio-token\") pod \"e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.416062 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.416025 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:25.764502 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.764474 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj"] Oct 29 23:20:25.765986 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:20:25.765953 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb26786c8_6181_41f2_a978_49b2123a690c.slice/crio-a9db1784f8a439904d4c7085743d8f619375c63ed7451bbc36a21a5db0e56e2e WatchSource:0}: Error finding container a9db1784f8a439904d4c7085743d8f619375c63ed7451bbc36a21a5db0e56e2e: Status 404 returned error can't find the container with id a9db1784f8a439904d4c7085743d8f619375c63ed7451bbc36a21a5db0e56e2e Oct 29 23:20:25.768972 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.768936 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:20:25.769064 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.769008 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:20:25.769064 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:25.769041 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:20:26.498766 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:26.498725 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" event={"ID":"b26786c8-6181-41f2-a978-49b2123a690c","Type":"ContainerStarted","Data":"f1eac2fd1b7c7c546f44f512c1bcd1d8c04780e3cf1f66c3b04e5f2a3db55990"} Oct 29 23:20:26.499426 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:26.499385 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" event={"ID":"b26786c8-6181-41f2-a978-49b2123a690c","Type":"ContainerStarted","Data":"a9db1784f8a439904d4c7085743d8f619375c63ed7451bbc36a21a5db0e56e2e"} Oct 29 23:20:26.552495 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:26.552431 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" podStartSLOduration=1.552412567 podStartE2EDuration="1.552412567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:20:26.549815356 +0000 UTC m=+2280.490363268" watchObservedRunningTime="2026-10-29 23:20:26.552412567 +0000 UTC m=+2280.492960477" Oct 29 23:20:27.423227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:27.418555 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:27.423455 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:27.423312 4821 patch_prober.go:29] interesting pod/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body= Oct 29 23:20:27.423455 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:27.423366 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" podUID="b26786c8-6181-41f2-a978-49b2123a690c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" Oct 29 23:20:28.419264 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:28.418620 4821 patch_prober.go:29] interesting pod/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body= Oct 29 23:20:28.419264 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:28.418680 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" podUID="b26786c8-6181-41f2-a978-49b2123a690c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" Oct 29 23:20:29.417676 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:29.417636 4821 patch_prober.go:29] interesting pod/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body= Oct 29 23:20:29.417992 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:29.417959 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" podUID="b26786c8-6181-41f2-a978-49b2123a690c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" Oct 29 23:20:30.276586 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:30.276557 4821 scope.go:122] "RemoveContainer" containerID="b7ec4b890c128073f9d18dea98a7454270445657c6a48a8ddd84c2041021a75d" Oct 29 23:20:30.439227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:30.438815 4821 patch_prober.go:29] interesting pod/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body= Oct 29 23:20:30.439227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:30.438877 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" podUID="b26786c8-6181-41f2-a978-49b2123a690c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" Oct 29 23:20:31.420230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:31.418569 4821 patch_prober.go:29] interesting pod/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body= Oct 29 23:20:31.420230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:31.418643 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" podUID="b26786c8-6181-41f2-a978-49b2123a690c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" Oct 29 23:20:32.417504 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:32.417462 4821 patch_prober.go:29] interesting pod/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body= Oct 29 23:20:32.417688 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:32.417519 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" podUID="b26786c8-6181-41f2-a978-49b2123a690c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" Oct 29 23:20:33.422545 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:33.422512 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:33.423020 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:33.422942 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:33.423517 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:33.423496 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:20:36.636410 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.636380 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj"] Oct 29 23:20:36.643518 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.643478 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:20:36.663462 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.663432 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj"] Oct 29 23:20:36.670389 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.670331 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-faa4fc\"/\"default-dockercfg-58lxt\"" Oct 29 23:20:36.670557 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.670383 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-faa4fc\"/\"test-model-faa4fc-kserve-self-signed-certs\"" Oct 29 23:20:36.673770 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.673750 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-faa4fc\"/\"openshift-service-ca.crt\"" Oct 29 23:20:36.673912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.673802 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-faa4fc\"/\"kube-root-ca.crt\"" Oct 29 23:20:36.742632 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.742602 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-dshm\") pod \"test-model-faa4fc-kserve-86f78b665c-ncfbj\" (UID: \"631ed840-42fc-45b1-92a3-b05f747bd17f\") " pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:20:36.742825 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.742802 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/631ed840-42fc-45b1-92a3-b05f747bd17f-tls-certs\") pod \"test-model-faa4fc-kserve-86f78b665c-ncfbj\" (UID: \"631ed840-42fc-45b1-92a3-b05f747bd17f\") " pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:20:36.743007 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.742989 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-home\") pod \"test-model-faa4fc-kserve-86f78b665c-ncfbj\" (UID: \"631ed840-42fc-45b1-92a3-b05f747bd17f\") " pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:20:36.743059 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.743023 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-model-cache\") pod \"test-model-faa4fc-kserve-86f78b665c-ncfbj\" (UID: \"631ed840-42fc-45b1-92a3-b05f747bd17f\") " pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:20:36.743178 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.743157 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-tmp-dir\") pod \"test-model-faa4fc-kserve-86f78b665c-ncfbj\" (UID: \"631ed840-42fc-45b1-92a3-b05f747bd17f\") " pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:20:36.743266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.743225 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8p78\" (UniqueName: \"kubernetes.io/projected/631ed840-42fc-45b1-92a3-b05f747bd17f-kube-api-access-k8p78\") pod \"test-model-faa4fc-kserve-86f78b665c-ncfbj\" (UID: \"631ed840-42fc-45b1-92a3-b05f747bd17f\") " pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:20:36.844986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.844938 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/631ed840-42fc-45b1-92a3-b05f747bd17f-tls-certs\") pod \"test-model-faa4fc-kserve-86f78b665c-ncfbj\" (UID: \"631ed840-42fc-45b1-92a3-b05f747bd17f\") " pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:20:36.845225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.845110 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-home\") pod \"test-model-faa4fc-kserve-86f78b665c-ncfbj\" (UID: \"631ed840-42fc-45b1-92a3-b05f747bd17f\") " pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:20:36.845225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.845136 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-model-cache\") pod \"test-model-faa4fc-kserve-86f78b665c-ncfbj\" (UID: \"631ed840-42fc-45b1-92a3-b05f747bd17f\") " pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:20:36.845225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.845215 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-tmp-dir\") pod \"test-model-faa4fc-kserve-86f78b665c-ncfbj\" (UID: \"631ed840-42fc-45b1-92a3-b05f747bd17f\") " pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:20:36.845419 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.845250 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8p78\" (UniqueName: \"kubernetes.io/projected/631ed840-42fc-45b1-92a3-b05f747bd17f-kube-api-access-k8p78\") pod \"test-model-faa4fc-kserve-86f78b665c-ncfbj\" (UID: \"631ed840-42fc-45b1-92a3-b05f747bd17f\") " pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:20:36.845419 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.845303 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-dshm\") pod \"test-model-faa4fc-kserve-86f78b665c-ncfbj\" (UID: \"631ed840-42fc-45b1-92a3-b05f747bd17f\") " pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:20:36.845601 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.845574 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-home\") pod \"test-model-faa4fc-kserve-86f78b665c-ncfbj\" (UID: \"631ed840-42fc-45b1-92a3-b05f747bd17f\") " pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:20:36.845668 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.845614 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-model-cache\") pod \"test-model-faa4fc-kserve-86f78b665c-ncfbj\" (UID: \"631ed840-42fc-45b1-92a3-b05f747bd17f\") " pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:20:36.845735 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.845665 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-tmp-dir\") pod \"test-model-faa4fc-kserve-86f78b665c-ncfbj\" (UID: \"631ed840-42fc-45b1-92a3-b05f747bd17f\") " pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:20:36.847915 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.847891 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-dshm\") pod \"test-model-faa4fc-kserve-86f78b665c-ncfbj\" (UID: \"631ed840-42fc-45b1-92a3-b05f747bd17f\") " pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:20:36.848342 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.848317 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/631ed840-42fc-45b1-92a3-b05f747bd17f-tls-certs\") pod \"test-model-faa4fc-kserve-86f78b665c-ncfbj\" (UID: \"631ed840-42fc-45b1-92a3-b05f747bd17f\") " pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:20:36.901274 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.898959 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8p78\" (UniqueName: \"kubernetes.io/projected/631ed840-42fc-45b1-92a3-b05f747bd17f-kube-api-access-k8p78\") pod \"test-model-faa4fc-kserve-86f78b665c-ncfbj\" (UID: \"631ed840-42fc-45b1-92a3-b05f747bd17f\") " pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:20:36.972243 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:36.967136 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:20:37.428995 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:37.428952 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj"] Oct 29 23:20:37.429333 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:20:37.429309 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod631ed840_42fc_45b1_92a3_b05f747bd17f.slice/crio-7f515e315a3a110fc3d2c898f90b6c9f973f5207ed2322d99c703d488c576451 WatchSource:0}: Error finding container 7f515e315a3a110fc3d2c898f90b6c9f973f5207ed2322d99c703d488c576451: Status 404 returned error can't find the container with id 7f515e315a3a110fc3d2c898f90b6c9f973f5207ed2322d99c703d488c576451 Oct 29 23:20:37.629139 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:37.629098 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" event={"ID":"631ed840-42fc-45b1-92a3-b05f747bd17f","Type":"ContainerStarted","Data":"2a493bf7019ae63fcff79e450733944d8c895258bd464cfbee7271e7b17748ff"} Oct 29 23:20:37.629139 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:37.629141 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" event={"ID":"631ed840-42fc-45b1-92a3-b05f747bd17f","Type":"ContainerStarted","Data":"7f515e315a3a110fc3d2c898f90b6c9f973f5207ed2322d99c703d488c576451"} Oct 29 23:20:37.677767 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:37.677674 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" podStartSLOduration=1.677659172 podStartE2EDuration="1.677659172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:20:37.672712124 +0000 UTC m=+2291.613260034" watchObservedRunningTime="2026-10-29 23:20:37.677659172 +0000 UTC m=+2291.618207081" Oct 29 23:20:37.972706 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:37.972632 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:20:46.973079 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:46.973045 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:20:46.985687 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:46.985655 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:20:47.740744 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:20:47.740712 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:20:53.890937 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:20:53.890895 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:21:02.661347 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:02.661311 4821 patch_prober.go:29] interesting pod/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:21:02.661811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:02.661387 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" podUID="9a986cc4-88d7-432a-8cc8-f2d5371373b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:21:10.283753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:10.283718 4821 generic.go:356] "Generic (PLEG): container finished" podID="9a986cc4-88d7-432a-8cc8-f2d5371373b1" containerID="c1cdf8455f07181d0ffbed0d991db9b77bbcef63bb87b23087abf6a61b0404ef" exitCode=0 Oct 29 23:21:10.284253 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:10.283797 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" event={"ID":"9a986cc4-88d7-432a-8cc8-f2d5371373b1","Type":"ContainerDied","Data":"c1cdf8455f07181d0ffbed0d991db9b77bbcef63bb87b23087abf6a61b0404ef"} Oct 29 23:21:10.284388 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:10.284368 4821 scope.go:122] "RemoveContainer" containerID="c1cdf8455f07181d0ffbed0d991db9b77bbcef63bb87b23087abf6a61b0404ef" Oct 29 23:21:10.286572 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:10.286541 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:21:10.286672 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:10.286611 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:21:10.286672 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:10.286642 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:21:11.294576 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:11.294535 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" event={"ID":"9a986cc4-88d7-432a-8cc8-f2d5371373b1","Type":"ContainerStarted","Data":"a804ec26806cf5bf0a1417b5334b75b0c170c846ddc48b4d62841ba035b2dccd"} Oct 29 23:21:11.586416 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:11.586309 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:21:11.586637 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:11.586612 4821 patch_prober.go:29] interesting pod/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Oct 29 23:21:11.586690 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:11.586669 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" podUID="9a986cc4-88d7-432a-8cc8-f2d5371373b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Oct 29 23:21:12.586231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:12.586169 4821 patch_prober.go:29] interesting pod/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Oct 29 23:21:12.586617 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:12.586264 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" podUID="9a986cc4-88d7-432a-8cc8-f2d5371373b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Oct 29 23:21:13.586424 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:13.586390 4821 patch_prober.go:29] interesting pod/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Oct 29 23:21:13.586855 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:13.586442 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" podUID="9a986cc4-88d7-432a-8cc8-f2d5371373b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Oct 29 23:21:14.587913 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:14.587836 4821 patch_prober.go:29] interesting pod/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Oct 29 23:21:14.589617 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:14.589575 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" podUID="9a986cc4-88d7-432a-8cc8-f2d5371373b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Oct 29 23:21:15.586023 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:15.585988 4821 patch_prober.go:29] interesting pod/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Oct 29 23:21:15.586253 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:15.586049 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" podUID="9a986cc4-88d7-432a-8cc8-f2d5371373b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Oct 29 23:21:16.596232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:16.588362 4821 patch_prober.go:29] interesting pod/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Oct 29 23:21:16.604239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:16.600283 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" podUID="9a986cc4-88d7-432a-8cc8-f2d5371373b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Oct 29 23:21:16.621576 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:16.621546 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:21:17.660157 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:17.660121 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:21:17.663289 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:17.663264 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:21:23.888427 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:21:23.888384 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:21:24.192190 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:24.192073 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg"] Oct 29 23:21:24.192414 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:24.192390 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" podUID="f275822e-4f33-4366-8be5-348dc70ebffd" containerName="istio-proxy" containerID="cri-o://ad7ff2d1b41691e6031661823937c2475751e52c32a59610ad4416e4b781d647" gracePeriod=30 Oct 29 23:21:29.492013 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.491986 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:21:29.530439 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.530407 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f275822e-4f33-4366-8be5-348dc70ebffd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f275822e-4f33-4366-8be5-348dc70ebffd-istio-podinfo\") pod \"f275822e-4f33-4366-8be5-348dc70ebffd\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " Oct 29 23:21:29.530599 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.530446 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-workload-certs\") pod \"f275822e-4f33-4366-8be5-348dc70ebffd\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " Oct 29 23:21:29.530599 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.530479 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f275822e-4f33-4366-8be5-348dc70ebffd-kube-api-access-r76vc\" (UniqueName: \"kubernetes.io/projected/f275822e-4f33-4366-8be5-348dc70ebffd-kube-api-access-r76vc\") pod \"f275822e-4f33-4366-8be5-348dc70ebffd\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " Oct 29 23:21:29.530599 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.530516 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-workload-socket\") pod \"f275822e-4f33-4366-8be5-348dc70ebffd\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " Oct 29 23:21:29.530746 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.530610 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-credential-socket\") pod \"f275822e-4f33-4366-8be5-348dc70ebffd\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " Oct 29 23:21:29.530746 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.530667 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-istio-envoy\") pod \"f275822e-4f33-4366-8be5-348dc70ebffd\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " Oct 29 23:21:29.530746 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.530715 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-workload-certs" pod "f275822e-4f33-4366-8be5-348dc70ebffd" (UID: "f275822e-4f33-4366-8be5-348dc70ebffd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:29.530746 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.530727 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-istio-data\") pod \"f275822e-4f33-4366-8be5-348dc70ebffd\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " Oct 29 23:21:29.530967 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.530913 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-workload-socket" pod "f275822e-4f33-4366-8be5-348dc70ebffd" (UID: "f275822e-4f33-4366-8be5-348dc70ebffd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:29.530967 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.530921 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f275822e-4f33-4366-8be5-348dc70ebffd-istio-token\" (UniqueName: \"kubernetes.io/projected/f275822e-4f33-4366-8be5-348dc70ebffd-istio-token\") pod \"f275822e-4f33-4366-8be5-348dc70ebffd\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " Oct 29 23:21:29.530967 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.530961 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-istio-data" pod "f275822e-4f33-4366-8be5-348dc70ebffd" (UID: "f275822e-4f33-4366-8be5-348dc70ebffd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:29.531137 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.530990 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-credential-socket" pod "f275822e-4f33-4366-8be5-348dc70ebffd" (UID: "f275822e-4f33-4366-8be5-348dc70ebffd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:29.531137 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.531072 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f275822e-4f33-4366-8be5-348dc70ebffd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f275822e-4f33-4366-8be5-348dc70ebffd-istiod-ca-cert\") pod \"f275822e-4f33-4366-8be5-348dc70ebffd\" (UID: \"f275822e-4f33-4366-8be5-348dc70ebffd\") " Oct 29 23:21:29.531532 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.531488 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f275822e-4f33-4366-8be5-348dc70ebffd-istiod-ca-cert" pod "f275822e-4f33-4366-8be5-348dc70ebffd" (UID: "f275822e-4f33-4366-8be5-348dc70ebffd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:21:29.532183 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.532160 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:21:29.532298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.532192 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f275822e-4f33-4366-8be5-348dc70ebffd-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:21:29.532298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.532225 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:21:29.532298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.532238 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:21:29.532298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.532253 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:21:29.533265 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.533240 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f275822e-4f33-4366-8be5-348dc70ebffd-kube-api-access-r76vc" pod "f275822e-4f33-4366-8be5-348dc70ebffd" (UID: "f275822e-4f33-4366-8be5-348dc70ebffd"). InnerVolumeSpecName "kube-api-access-r76vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:29.533397 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.533380 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f275822e-4f33-4366-8be5-348dc70ebffd-istio-token" pod "f275822e-4f33-4366-8be5-348dc70ebffd" (UID: "f275822e-4f33-4366-8be5-348dc70ebffd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:21:29.533506 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.533487 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f275822e-4f33-4366-8be5-348dc70ebffd-istio-podinfo" pod "f275822e-4f33-4366-8be5-348dc70ebffd" (UID: "f275822e-4f33-4366-8be5-348dc70ebffd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:21:29.533551 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.533497 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-istio-envoy" pod "f275822e-4f33-4366-8be5-348dc70ebffd" (UID: "f275822e-4f33-4366-8be5-348dc70ebffd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:21:29.600771 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.600735 4821 generic.go:356] "Generic (PLEG): container finished" podID="f275822e-4f33-4366-8be5-348dc70ebffd" containerID="ad7ff2d1b41691e6031661823937c2475751e52c32a59610ad4416e4b781d647" exitCode=0 Oct 29 23:21:29.600959 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.600792 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" Oct 29 23:21:29.600959 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.600819 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" event={"ID":"f275822e-4f33-4366-8be5-348dc70ebffd","Type":"ContainerDied","Data":"ad7ff2d1b41691e6031661823937c2475751e52c32a59610ad4416e4b781d647"} Oct 29 23:21:29.600959 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.600882 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg" event={"ID":"f275822e-4f33-4366-8be5-348dc70ebffd","Type":"ContainerDied","Data":"aba5149732889a4a9e1aa53426fbf46357bd4e617ba527d000b8c8d388a7c5f9"} Oct 29 23:21:29.600959 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.600910 4821 scope.go:122] "RemoveContainer" containerID="ad7ff2d1b41691e6031661823937c2475751e52c32a59610ad4416e4b781d647" Oct 29 23:21:29.616783 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.616761 4821 scope.go:122] "RemoveContainer" containerID="19e90a2a30dd1b2974e6f079979fea2d04f67623ea7fe4eafb908ebb571244b9" Oct 29 23:21:29.627595 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.627576 4821 scope.go:122] "RemoveContainer" containerID="ad7ff2d1b41691e6031661823937c2475751e52c32a59610ad4416e4b781d647" Oct 29 23:21:29.627867 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:21:29.627845 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad7ff2d1b41691e6031661823937c2475751e52c32a59610ad4416e4b781d647\": container with ID starting with ad7ff2d1b41691e6031661823937c2475751e52c32a59610ad4416e4b781d647 not found: ID does not exist" containerID="ad7ff2d1b41691e6031661823937c2475751e52c32a59610ad4416e4b781d647" Oct 29 23:21:29.627951 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.627876 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad7ff2d1b41691e6031661823937c2475751e52c32a59610ad4416e4b781d647"} err="failed to get container status \"ad7ff2d1b41691e6031661823937c2475751e52c32a59610ad4416e4b781d647\": rpc error: code = NotFound desc = could not find container \"ad7ff2d1b41691e6031661823937c2475751e52c32a59610ad4416e4b781d647\": container with ID starting with ad7ff2d1b41691e6031661823937c2475751e52c32a59610ad4416e4b781d647 not found: ID does not exist" Oct 29 23:21:29.627951 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.627895 4821 scope.go:122] "RemoveContainer" containerID="19e90a2a30dd1b2974e6f079979fea2d04f67623ea7fe4eafb908ebb571244b9" Oct 29 23:21:29.628211 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:21:29.628178 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19e90a2a30dd1b2974e6f079979fea2d04f67623ea7fe4eafb908ebb571244b9\": container with ID starting with 19e90a2a30dd1b2974e6f079979fea2d04f67623ea7fe4eafb908ebb571244b9 not found: ID does not exist" containerID="19e90a2a30dd1b2974e6f079979fea2d04f67623ea7fe4eafb908ebb571244b9" Oct 29 23:21:29.628272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.628219 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e90a2a30dd1b2974e6f079979fea2d04f67623ea7fe4eafb908ebb571244b9"} err="failed to get container status \"19e90a2a30dd1b2974e6f079979fea2d04f67623ea7fe4eafb908ebb571244b9\": rpc error: code = NotFound desc = could not find container \"19e90a2a30dd1b2974e6f079979fea2d04f67623ea7fe4eafb908ebb571244b9\": container with ID starting with 19e90a2a30dd1b2974e6f079979fea2d04f67623ea7fe4eafb908ebb571244b9 not found: ID does not exist" Oct 29 23:21:29.634141 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.634121 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f275822e-4f33-4366-8be5-348dc70ebffd-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:21:29.634246 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.634143 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r76vc\" (UniqueName: \"kubernetes.io/projected/f275822e-4f33-4366-8be5-348dc70ebffd-kube-api-access-r76vc\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:21:29.634246 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.634153 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f275822e-4f33-4366-8be5-348dc70ebffd-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:21:29.634246 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.634163 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f275822e-4f33-4366-8be5-348dc70ebffd-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:21:29.659627 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.659601 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg"] Oct 29 23:21:29.664013 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:29.663990 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-49d87f-openshift-default-7bb5bbd88f-dbxfg"] Oct 29 23:21:30.173160 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:30.173128 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f275822e-4f33-4366-8be5-348dc70ebffd" path="/var/lib/kubelet/pods/f275822e-4f33-4366-8be5-348dc70ebffd/volumes" Oct 29 23:21:30.352161 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:30.352130 4821 scope.go:122] "RemoveContainer" containerID="0d2f4f9130a6858a01ecdb0438ec1af39f7e003b07fdf40f644832e4aec04d2d" Oct 29 23:21:30.363195 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:30.363173 4821 scope.go:122] "RemoveContainer" containerID="6b77984258c52ea3cd87ac5ecbaf4b39967292b7a6cb556516084b01aca652d7" Oct 29 23:21:30.373067 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:30.373041 4821 scope.go:122] "RemoveContainer" containerID="f27e133cecacd9a539a2648aa5caf3d2e2b58d1f047ed238d8b062904f4caca7" Oct 29 23:21:54.101224 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:21:54.101077 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:21:56.006350 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:56.006321 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc"] Oct 29 23:21:56.010042 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:21:56.010009 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" podUID="fd8d67b9-8725-4c23-a125-9bfdf3ff4917" containerName="istio-proxy" containerID="cri-o://313a4ab372e3140a6d4933d354e7c6a35cfc98a8c03942011d79430fbaf6940a" gracePeriod=30 Oct 29 23:22:00.272697 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:00.272664 4821 patch_prober.go:29] interesting pod/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:22:00.273087 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:00.272726 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" podUID="fd8d67b9-8725-4c23-a125-9bfdf3ff4917" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:22:01.306699 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.306671 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 23:22:01.389447 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.389411 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-workload-certs\") pod \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " Oct 29 23:22:01.389666 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.389501 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-token\" (UniqueName: \"kubernetes.io/projected/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-token\") pod \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " Oct 29 23:22:01.389666 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.389543 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istiod-ca-cert\") pod \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " Oct 29 23:22:01.389666 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.389590 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-workload-socket\") pod \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " Oct 29 23:22:01.389666 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.389652 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-podinfo\") pod \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " Oct 29 23:22:01.389867 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.389723 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-workload-certs" pod "fd8d67b9-8725-4c23-a125-9bfdf3ff4917" (UID: "fd8d67b9-8725-4c23-a125-9bfdf3ff4917"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:01.389867 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.389746 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-data\") pod \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " Oct 29 23:22:01.389951 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.389865 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-credential-socket\") pod \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " Oct 29 23:22:01.390019 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.389959 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-envoy\") pod \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " Oct 29 23:22:01.390019 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.390010 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-kube-api-access-crpl7\" (UniqueName: \"kubernetes.io/projected/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-kube-api-access-crpl7\") pod \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\" (UID: \"fd8d67b9-8725-4c23-a125-9bfdf3ff4917\") " Oct 29 23:22:01.390120 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.390044 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-data" pod "fd8d67b9-8725-4c23-a125-9bfdf3ff4917" (UID: "fd8d67b9-8725-4c23-a125-9bfdf3ff4917"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:01.390187 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.390165 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-workload-socket" pod "fd8d67b9-8725-4c23-a125-9bfdf3ff4917" (UID: "fd8d67b9-8725-4c23-a125-9bfdf3ff4917"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:01.390275 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.390187 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-credential-socket" pod "fd8d67b9-8725-4c23-a125-9bfdf3ff4917" (UID: "fd8d67b9-8725-4c23-a125-9bfdf3ff4917"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:01.390932 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.390907 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:22:01.391075 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.390937 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:22:01.391075 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.390951 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:22:01.391075 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.390964 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:22:01.391075 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.390905 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istiod-ca-cert" pod "fd8d67b9-8725-4c23-a125-9bfdf3ff4917" (UID: "fd8d67b9-8725-4c23-a125-9bfdf3ff4917"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:22:01.393240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.392427 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-token" pod "fd8d67b9-8725-4c23-a125-9bfdf3ff4917" (UID: "fd8d67b9-8725-4c23-a125-9bfdf3ff4917"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:22:01.393240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.392610 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-podinfo" pod "fd8d67b9-8725-4c23-a125-9bfdf3ff4917" (UID: "fd8d67b9-8725-4c23-a125-9bfdf3ff4917"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:22:01.393589 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.393570 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-envoy" pod "fd8d67b9-8725-4c23-a125-9bfdf3ff4917" (UID: "fd8d67b9-8725-4c23-a125-9bfdf3ff4917"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:22:01.393659 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.393645 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-kube-api-access-crpl7" pod "fd8d67b9-8725-4c23-a125-9bfdf3ff4917" (UID: "fd8d67b9-8725-4c23-a125-9bfdf3ff4917"). InnerVolumeSpecName "kube-api-access-crpl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:22:01.492889 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.492857 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:22:01.492889 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.492886 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:22:01.492889 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.492895 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:22:01.493136 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.492903 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:22:01.493136 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:01.492911 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crpl7\" (UniqueName: \"kubernetes.io/projected/fd8d67b9-8725-4c23-a125-9bfdf3ff4917-kube-api-access-crpl7\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:22:02.082112 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:02.082079 4821 generic.go:356] "Generic (PLEG): container finished" podID="fd8d67b9-8725-4c23-a125-9bfdf3ff4917" containerID="313a4ab372e3140a6d4933d354e7c6a35cfc98a8c03942011d79430fbaf6940a" exitCode=0 Oct 29 23:22:02.082112 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:02.082116 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" event={"ID":"fd8d67b9-8725-4c23-a125-9bfdf3ff4917","Type":"ContainerDied","Data":"313a4ab372e3140a6d4933d354e7c6a35cfc98a8c03942011d79430fbaf6940a"} Oct 29 23:22:02.082435 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:02.082131 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" Oct 29 23:22:02.082435 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:02.082154 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc" event={"ID":"fd8d67b9-8725-4c23-a125-9bfdf3ff4917","Type":"ContainerDied","Data":"056aeb5289ac31cee7c3aa6b1b2ea48330a78a7f3aba1bced9d538ad93b9ad51"} Oct 29 23:22:02.082435 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:02.082174 4821 scope.go:122] "RemoveContainer" containerID="313a4ab372e3140a6d4933d354e7c6a35cfc98a8c03942011d79430fbaf6940a" Oct 29 23:22:02.095128 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:02.095101 4821 scope.go:122] "RemoveContainer" containerID="313a4ab372e3140a6d4933d354e7c6a35cfc98a8c03942011d79430fbaf6940a" Oct 29 23:22:02.095519 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:22:02.095486 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313a4ab372e3140a6d4933d354e7c6a35cfc98a8c03942011d79430fbaf6940a\": container with ID starting with 313a4ab372e3140a6d4933d354e7c6a35cfc98a8c03942011d79430fbaf6940a not found: ID does not exist" containerID="313a4ab372e3140a6d4933d354e7c6a35cfc98a8c03942011d79430fbaf6940a" Oct 29 23:22:02.095624 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:02.095520 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313a4ab372e3140a6d4933d354e7c6a35cfc98a8c03942011d79430fbaf6940a"} err="failed to get container status \"313a4ab372e3140a6d4933d354e7c6a35cfc98a8c03942011d79430fbaf6940a\": rpc error: code = NotFound desc = could not find container \"313a4ab372e3140a6d4933d354e7c6a35cfc98a8c03942011d79430fbaf6940a\": container with ID starting with 313a4ab372e3140a6d4933d354e7c6a35cfc98a8c03942011d79430fbaf6940a not found: ID does not exist" Oct 29 23:22:02.180154 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:02.180119 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc"] Oct 29 23:22:02.193559 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:02.193534 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-x55bc"] Oct 29 23:22:04.217319 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:04.217282 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd8d67b9-8725-4c23-a125-9bfdf3ff4917" path="/var/lib/kubelet/pods/fd8d67b9-8725-4c23-a125-9bfdf3ff4917/volumes" Oct 29 23:22:17.125432 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.125392 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg"] Oct 29 23:22:17.126473 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.126440 4821 cpu_manager.go:414] "containerMap: removing container" podUID="f275822e-4f33-4366-8be5-348dc70ebffd" containerName="istio-proxy" Oct 29 23:22:17.126473 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.126464 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f275822e-4f33-4366-8be5-348dc70ebffd" containerName="istio-proxy" Oct 29 23:22:17.126709 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.126484 4821 cpu_manager.go:414] "containerMap: removing container" podUID="fd8d67b9-8725-4c23-a125-9bfdf3ff4917" containerName="istio-proxy" Oct 29 23:22:17.126709 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.126492 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd8d67b9-8725-4c23-a125-9bfdf3ff4917" containerName="istio-proxy" Oct 29 23:22:17.126709 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.126535 4821 cpu_manager.go:414] "containerMap: removing container" podUID="f275822e-4f33-4366-8be5-348dc70ebffd" containerName="istio-proxy" Oct 29 23:22:17.126709 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.126544 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f275822e-4f33-4366-8be5-348dc70ebffd" containerName="istio-proxy" Oct 29 23:22:17.126925 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.126735 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="f275822e-4f33-4366-8be5-348dc70ebffd" containerName="istio-proxy" Oct 29 23:22:17.126925 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.126753 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd8d67b9-8725-4c23-a125-9bfdf3ff4917" containerName="istio-proxy" Oct 29 23:22:17.132162 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.132142 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.169772 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.169743 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg"] Oct 29 23:22:17.246341 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.246302 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e871ef2-d293-4237-b33c-996d7637d5c1-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.246533 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.246392 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e871ef2-d293-4237-b33c-996d7637d5c1-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.246671 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.246648 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e871ef2-d293-4237-b33c-996d7637d5c1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.246730 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.246679 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e871ef2-d293-4237-b33c-996d7637d5c1-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.246775 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.246730 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e871ef2-d293-4237-b33c-996d7637d5c1-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.246903 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.246888 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e871ef2-d293-4237-b33c-996d7637d5c1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.246986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.246956 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e871ef2-d293-4237-b33c-996d7637d5c1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.247050 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.247034 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkfrv\" (UniqueName: \"kubernetes.io/projected/0e871ef2-d293-4237-b33c-996d7637d5c1-kube-api-access-qkfrv\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.247091 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.247083 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e871ef2-d293-4237-b33c-996d7637d5c1-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.352714 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.349756 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e871ef2-d293-4237-b33c-996d7637d5c1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.353032 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.353000 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e871ef2-d293-4237-b33c-996d7637d5c1-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.353191 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.353165 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e871ef2-d293-4237-b33c-996d7637d5c1-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.353366 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.353246 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e871ef2-d293-4237-b33c-996d7637d5c1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.353366 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.353318 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e871ef2-d293-4237-b33c-996d7637d5c1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.353366 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.353362 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkfrv\" (UniqueName: \"kubernetes.io/projected/0e871ef2-d293-4237-b33c-996d7637d5c1-kube-api-access-qkfrv\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.353563 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.353429 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e871ef2-d293-4237-b33c-996d7637d5c1-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.353563 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.353540 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e871ef2-d293-4237-b33c-996d7637d5c1-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.353685 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.353621 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e871ef2-d293-4237-b33c-996d7637d5c1-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.356819 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.354239 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e871ef2-d293-4237-b33c-996d7637d5c1-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.357984 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.357930 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e871ef2-d293-4237-b33c-996d7637d5c1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.358450 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.358413 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e871ef2-d293-4237-b33c-996d7637d5c1-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.358836 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.358807 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e871ef2-d293-4237-b33c-996d7637d5c1-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.359551 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.359525 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e871ef2-d293-4237-b33c-996d7637d5c1-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.360325 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.360297 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e871ef2-d293-4237-b33c-996d7637d5c1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.369945 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.369891 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e871ef2-d293-4237-b33c-996d7637d5c1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.383911 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.383845 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkfrv\" (UniqueName: \"kubernetes.io/projected/0e871ef2-d293-4237-b33c-996d7637d5c1-kube-api-access-qkfrv\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.384047 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.383966 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e871ef2-d293-4237-b33c-996d7637d5c1-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-78xhg\" (UID: \"0e871ef2-d293-4237-b33c-996d7637d5c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.449652 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.449617 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:17.798638 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:22:17.798603 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e871ef2_d293_4237_b33c_996d7637d5c1.slice/crio-b8955725a1888a1693d77f0e102e02bdf382b1c2d6ec13f3079c5eb644249317 WatchSource:0}: Error finding container b8955725a1888a1693d77f0e102e02bdf382b1c2d6ec13f3079c5eb644249317: Status 404 returned error can't find the container with id b8955725a1888a1693d77f0e102e02bdf382b1c2d6ec13f3079c5eb644249317 Oct 29 23:22:17.799010 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.798989 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg"] Oct 29 23:22:17.801740 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.801710 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:22:17.801831 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.801777 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:22:17.801831 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:17.801805 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:22:18.514066 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:18.514026 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" event={"ID":"0e871ef2-d293-4237-b33c-996d7637d5c1","Type":"ContainerStarted","Data":"562ad028bd33086dd68f6a2216b18e217cabd3cfe53315d564b014e6c4d1edf6"} Oct 29 23:22:18.514497 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:18.514093 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" event={"ID":"0e871ef2-d293-4237-b33c-996d7637d5c1","Type":"ContainerStarted","Data":"b8955725a1888a1693d77f0e102e02bdf382b1c2d6ec13f3079c5eb644249317"} Oct 29 23:22:18.591617 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:18.591560 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" podStartSLOduration=1.591545314 podStartE2EDuration="1.591545314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:18.5763112 +0000 UTC m=+2392.516859110" watchObservedRunningTime="2026-10-29 23:22:18.591545314 +0000 UTC m=+2392.532093311" Oct 29 23:22:19.449868 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:19.449822 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:19.452428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:19.452086 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-78xhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body= Oct 29 23:22:19.452428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:19.452137 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" podUID="0e871ef2-d293-4237-b33c-996d7637d5c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" Oct 29 23:22:20.450358 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:20.450323 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-78xhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body= Oct 29 23:22:20.450781 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:20.450376 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" podUID="0e871ef2-d293-4237-b33c-996d7637d5c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" Oct 29 23:22:21.473243 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:21.471646 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-78xhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body= Oct 29 23:22:21.476307 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:21.476273 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" podUID="0e871ef2-d293-4237-b33c-996d7637d5c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" Oct 29 23:22:22.450052 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:22.450012 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-78xhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body= Oct 29 23:22:22.450289 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:22.450093 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" podUID="0e871ef2-d293-4237-b33c-996d7637d5c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" Oct 29 23:22:23.486484 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:23.486442 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-78xhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body= Oct 29 23:22:23.487481 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:23.487355 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" podUID="0e871ef2-d293-4237-b33c-996d7637d5c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" Oct 29 23:22:23.889671 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:22:23.889625 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:22:24.450693 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:24.450658 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-78xhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body= Oct 29 23:22:24.450906 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:24.450720 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" podUID="0e871ef2-d293-4237-b33c-996d7637d5c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" Oct 29 23:22:25.450103 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:25.450057 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-78xhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body= Oct 29 23:22:25.450642 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:25.450123 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" podUID="0e871ef2-d293-4237-b33c-996d7637d5c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" Oct 29 23:22:27.327276 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:27.327240 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:27.327748 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:27.327732 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:27.328619 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:27.328597 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-78xhg" Oct 29 23:22:32.148474 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.148437 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw"] Oct 29 23:22:32.149463 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.149441 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="f275822e-4f33-4366-8be5-348dc70ebffd" containerName="istio-proxy" Oct 29 23:22:32.154718 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.154697 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.185016 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.184979 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw"] Oct 29 23:22:32.197697 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.197665 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.197884 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.197822 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.197965 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.197932 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.198073 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.198058 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/084d3c77-3d39-4234-b601-df4b9226f135-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.198194 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.198175 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/084d3c77-3d39-4234-b601-df4b9226f135-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.198324 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.198268 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/084d3c77-3d39-4234-b601-df4b9226f135-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.198472 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.198449 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.198526 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.198486 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.198568 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.198522 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsq7t\" (UniqueName: \"kubernetes.io/projected/084d3c77-3d39-4234-b601-df4b9226f135-kube-api-access-qsq7t\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.300104 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.300072 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.300104 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.300109 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.300417 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.300246 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsq7t\" (UniqueName: \"kubernetes.io/projected/084d3c77-3d39-4234-b601-df4b9226f135-kube-api-access-qsq7t\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.300417 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.300316 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.300550 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.300419 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.300550 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.300449 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.300550 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.300466 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.300718 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.300598 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/084d3c77-3d39-4234-b601-df4b9226f135-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.300718 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.300647 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.300718 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.300695 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.300879 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.300741 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/084d3c77-3d39-4234-b601-df4b9226f135-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.300879 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.300779 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/084d3c77-3d39-4234-b601-df4b9226f135-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.300879 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.300787 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.301485 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.301458 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/084d3c77-3d39-4234-b601-df4b9226f135-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.302688 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.302669 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.303331 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.303309 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/084d3c77-3d39-4234-b601-df4b9226f135-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.324774 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.324746 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/084d3c77-3d39-4234-b601-df4b9226f135-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.324903 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.324888 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsq7t\" (UniqueName: \"kubernetes.io/projected/084d3c77-3d39-4234-b601-df4b9226f135-kube-api-access-qsq7t\") pod \"maas-default-gateway-openshift-default-6db586dd6c-djzhw\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.483762 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.483688 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:32.815637 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:22:32.815604 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084d3c77_3d39_4234_b601_df4b9226f135.slice/crio-d7ab1a727d5f71e71364f97ce11dec953a1960842aab0a63117e010b0e49f20d WatchSource:0}: Error finding container d7ab1a727d5f71e71364f97ce11dec953a1960842aab0a63117e010b0e49f20d: Status 404 returned error can't find the container with id d7ab1a727d5f71e71364f97ce11dec953a1960842aab0a63117e010b0e49f20d Oct 29 23:22:32.816055 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.816031 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw"] Oct 29 23:22:32.818731 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.818695 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:22:32.818843 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.818763 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:22:32.818843 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:32.818796 4821 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 29 23:22:33.713471 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:33.713435 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" event={"ID":"084d3c77-3d39-4234-b601-df4b9226f135","Type":"ContainerStarted","Data":"da8f801a56cb07cb1063ef06fd06d4a7c0b1e584e814c639d72cb374c82e8586"} Oct 29 23:22:33.713909 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:33.713498 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" event={"ID":"084d3c77-3d39-4234-b601-df4b9226f135","Type":"ContainerStarted","Data":"d7ab1a727d5f71e71364f97ce11dec953a1960842aab0a63117e010b0e49f20d"} Oct 29 23:22:33.800520 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:33.800466 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" podStartSLOduration=1.800450511 podStartE2EDuration="1.800450511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:33.784032065 +0000 UTC m=+2407.724579975" watchObservedRunningTime="2026-10-29 23:22:33.800450511 +0000 UTC m=+2407.740998421" Oct 29 23:22:34.483792 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:34.483755 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:34.485888 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:34.485858 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-djzhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body= Oct 29 23:22:34.486025 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:34.485923 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" podUID="084d3c77-3d39-4234-b601-df4b9226f135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" Oct 29 23:22:35.484623 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:35.484577 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-djzhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body= Oct 29 23:22:35.485095 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:35.484717 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" podUID="084d3c77-3d39-4234-b601-df4b9226f135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" Oct 29 23:22:36.484573 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:36.484540 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-djzhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body= Oct 29 23:22:36.484737 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:36.484598 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" podUID="084d3c77-3d39-4234-b601-df4b9226f135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" Oct 29 23:22:37.484586 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:37.484549 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-djzhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body= Oct 29 23:22:37.484781 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:37.484611 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" podUID="084d3c77-3d39-4234-b601-df4b9226f135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" Oct 29 23:22:38.484787 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:38.484751 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-djzhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body= Oct 29 23:22:38.485246 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:38.484814 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" podUID="084d3c77-3d39-4234-b601-df4b9226f135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" Oct 29 23:22:39.484895 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:39.484862 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-djzhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body= Oct 29 23:22:39.485289 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:39.484923 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" podUID="084d3c77-3d39-4234-b601-df4b9226f135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" Oct 29 23:22:40.484106 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:40.484072 4821 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-djzhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body= Oct 29 23:22:40.484382 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:40.484140 4821 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" podUID="084d3c77-3d39-4234-b601-df4b9226f135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" Oct 29 23:22:41.727969 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:41.727934 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:41.728498 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:41.728445 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:41.729830 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:41.729805 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:22:44.205979 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.205942 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm"] Oct 29 23:22:44.212374 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.212345 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:22:44.235531 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.235495 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm"] Oct 29 23:22:44.242763 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.242735 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-dcacb9\"/\"kube-root-ca.crt\"" Oct 29 23:22:44.242947 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.242773 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-dcacb9\"/\"openshift-service-ca.crt\"" Oct 29 23:22:44.243082 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.243060 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-dcacb9\"/\"default-dockercfg-25nwp\"" Oct 29 23:22:44.243528 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.242889 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-dcacb9\"/\"test-model-dcacb9-kserve-self-signed-certs\"" Oct 29 23:22:44.279643 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.279602 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-home\") pod \"test-model-dcacb9-kserve-5dfb8c5df9-s2tjm\" (UID: \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\") " pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:22:44.279962 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.279931 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-tls-certs\") pod \"test-model-dcacb9-kserve-5dfb8c5df9-s2tjm\" (UID: \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\") " pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:22:44.280130 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.280008 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-tmp-dir\") pod \"test-model-dcacb9-kserve-5dfb8c5df9-s2tjm\" (UID: \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\") " pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:22:44.280130 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.280103 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vlqz\" (UniqueName: \"kubernetes.io/projected/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-kube-api-access-7vlqz\") pod \"test-model-dcacb9-kserve-5dfb8c5df9-s2tjm\" (UID: \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\") " pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:22:44.280417 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.280231 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-model-cache\") pod \"test-model-dcacb9-kserve-5dfb8c5df9-s2tjm\" (UID: \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\") " pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:22:44.280417 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.280315 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-dshm\") pod \"test-model-dcacb9-kserve-5dfb8c5df9-s2tjm\" (UID: \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\") " pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:22:44.382477 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.382443 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vlqz\" (UniqueName: \"kubernetes.io/projected/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-kube-api-access-7vlqz\") pod \"test-model-dcacb9-kserve-5dfb8c5df9-s2tjm\" (UID: \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\") " pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:22:44.382835 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.382810 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-model-cache\") pod \"test-model-dcacb9-kserve-5dfb8c5df9-s2tjm\" (UID: \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\") " pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:22:44.382975 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.382927 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-dshm\") pod \"test-model-dcacb9-kserve-5dfb8c5df9-s2tjm\" (UID: \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\") " pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:22:44.383095 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.383077 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-home\") pod \"test-model-dcacb9-kserve-5dfb8c5df9-s2tjm\" (UID: \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\") " pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:22:44.383182 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.383109 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-tls-certs\") pod \"test-model-dcacb9-kserve-5dfb8c5df9-s2tjm\" (UID: \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\") " pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:22:44.383315 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.383293 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-model-cache\") pod \"test-model-dcacb9-kserve-5dfb8c5df9-s2tjm\" (UID: \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\") " pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:22:44.383408 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.383368 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-tmp-dir\") pod \"test-model-dcacb9-kserve-5dfb8c5df9-s2tjm\" (UID: \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\") " pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:22:44.383479 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.383441 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-home\") pod \"test-model-dcacb9-kserve-5dfb8c5df9-s2tjm\" (UID: \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\") " pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:22:44.383715 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.383691 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-tmp-dir\") pod \"test-model-dcacb9-kserve-5dfb8c5df9-s2tjm\" (UID: \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\") " pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:22:44.385553 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.385533 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-dshm\") pod \"test-model-dcacb9-kserve-5dfb8c5df9-s2tjm\" (UID: \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\") " pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:22:44.385899 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.385878 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-tls-certs\") pod \"test-model-dcacb9-kserve-5dfb8c5df9-s2tjm\" (UID: \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\") " pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:22:44.407636 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.407614 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vlqz\" (UniqueName: \"kubernetes.io/projected/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-kube-api-access-7vlqz\") pod \"test-model-dcacb9-kserve-5dfb8c5df9-s2tjm\" (UID: \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\") " pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:22:44.530002 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.528680 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:22:44.971105 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:44.971076 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm"] Oct 29 23:22:44.971292 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:22:44.971180 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdb0e241_e40d_4cae_ab4c_1862ebc8dc9a.slice/crio-7005c4b5f1c0bafde691df63e44cc302fe55aa8fe52c755150a4e7f1497931db WatchSource:0}: Error finding container 7005c4b5f1c0bafde691df63e44cc302fe55aa8fe52c755150a4e7f1497931db: Status 404 returned error can't find the container with id 7005c4b5f1c0bafde691df63e44cc302fe55aa8fe52c755150a4e7f1497931db Oct 29 23:22:45.851182 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:45.851138 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" event={"ID":"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a","Type":"ContainerStarted","Data":"44e235dcf62f1877e02cc0cce5db59c0a3e0107fe6f592e0d115df5b63704cf3"} Oct 29 23:22:45.851728 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:45.851191 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" event={"ID":"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a","Type":"ContainerStarted","Data":"7005c4b5f1c0bafde691df63e44cc302fe55aa8fe52c755150a4e7f1497931db"} Oct 29 23:22:45.898742 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:45.898681 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" podStartSLOduration=1.8986620570000001 podStartE2EDuration="1.898662057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:45.89487459 +0000 UTC m=+2419.835422501" watchObservedRunningTime="2026-10-29 23:22:45.898662057 +0000 UTC m=+2419.839209968" Oct 29 23:22:46.530369 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:46.530338 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:22:47.325592 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.325554 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49"] Oct 29 23:22:47.330869 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.330842 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:22:47.354997 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.354966 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49"] Oct 29 23:22:47.361613 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.361565 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cc6328\"/\"kube-root-ca.crt\"" Oct 29 23:22:47.361781 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.361665 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cc6328\"/\"openshift-service-ca.crt\"" Oct 29 23:22:47.365840 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.365818 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cc6328\"/\"default-dockercfg-tfzkh\"" Oct 29 23:22:47.365840 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.365817 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cc6328\"/\"test-model-cc6328-kserve-self-signed-certs\"" Oct 29 23:22:47.393067 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.393035 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-tls-certs\") pod \"test-model-cc6328-kserve-65d5d97477-h4b49\" (UID: \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\") " pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:22:47.393274 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.393253 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-dshm\") pod \"test-model-cc6328-kserve-65d5d97477-h4b49\" (UID: \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\") " pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:22:47.393426 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.393401 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf69t\" (UniqueName: \"kubernetes.io/projected/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-kube-api-access-kf69t\") pod \"test-model-cc6328-kserve-65d5d97477-h4b49\" (UID: \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\") " pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:22:47.393540 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.393522 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-model-cache\") pod \"test-model-cc6328-kserve-65d5d97477-h4b49\" (UID: \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\") " pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:22:47.393600 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.393582 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-tmp-dir\") pod \"test-model-cc6328-kserve-65d5d97477-h4b49\" (UID: \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\") " pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:22:47.393678 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.393660 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-home\") pod \"test-model-cc6328-kserve-65d5d97477-h4b49\" (UID: \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\") " pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:22:47.495828 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.495783 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-dshm\") pod \"test-model-cc6328-kserve-65d5d97477-h4b49\" (UID: \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\") " pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:22:47.496041 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.495900 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf69t\" (UniqueName: \"kubernetes.io/projected/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-kube-api-access-kf69t\") pod \"test-model-cc6328-kserve-65d5d97477-h4b49\" (UID: \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\") " pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:22:47.496041 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.495981 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-model-cache\") pod \"test-model-cc6328-kserve-65d5d97477-h4b49\" (UID: \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\") " pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:22:47.496180 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.496144 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-tmp-dir\") pod \"test-model-cc6328-kserve-65d5d97477-h4b49\" (UID: \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\") " pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:22:47.496313 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.496243 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-home\") pod \"test-model-cc6328-kserve-65d5d97477-h4b49\" (UID: \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\") " pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:22:47.496459 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.496437 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-model-cache\") pod \"test-model-cc6328-kserve-65d5d97477-h4b49\" (UID: \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\") " pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:22:47.496544 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.496488 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-tmp-dir\") pod \"test-model-cc6328-kserve-65d5d97477-h4b49\" (UID: \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\") " pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:22:47.496544 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.496511 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-tls-certs\") pod \"test-model-cc6328-kserve-65d5d97477-h4b49\" (UID: \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\") " pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:22:47.496679 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.496660 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-home\") pod \"test-model-cc6328-kserve-65d5d97477-h4b49\" (UID: \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\") " pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:22:47.498969 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.498944 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-dshm\") pod \"test-model-cc6328-kserve-65d5d97477-h4b49\" (UID: \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\") " pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:22:47.499110 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.499094 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-tls-certs\") pod \"test-model-cc6328-kserve-65d5d97477-h4b49\" (UID: \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\") " pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:22:47.521024 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.520988 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf69t\" (UniqueName: \"kubernetes.io/projected/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-kube-api-access-kf69t\") pod \"test-model-cc6328-kserve-65d5d97477-h4b49\" (UID: \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\") " pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:22:47.655237 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:47.653444 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:22:48.144019 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:22:48.143989 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99a4a181_f0c4_438e_a5a3_f8764e83cfaf.slice/crio-52e0e01fe455c84299ad08f93d5b5817abb7a0d26f1fd8c663a369bd34c9145d WatchSource:0}: Error finding container 52e0e01fe455c84299ad08f93d5b5817abb7a0d26f1fd8c663a369bd34c9145d: Status 404 returned error can't find the container with id 52e0e01fe455c84299ad08f93d5b5817abb7a0d26f1fd8c663a369bd34c9145d Oct 29 23:22:48.144157 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:48.144128 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49"] Oct 29 23:22:48.885987 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:48.885950 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" event={"ID":"99a4a181-f0c4-438e-a5a3-f8764e83cfaf","Type":"ContainerStarted","Data":"45fc73318a92e96e81e988c7723fcd2f1700c4fa05455a5bbd47c9d5d901e0f8"} Oct 29 23:22:48.886414 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:48.885996 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" event={"ID":"99a4a181-f0c4-438e-a5a3-f8764e83cfaf","Type":"ContainerStarted","Data":"52e0e01fe455c84299ad08f93d5b5817abb7a0d26f1fd8c663a369bd34c9145d"} Oct 29 23:22:48.933485 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:48.933433 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" podStartSLOduration=1.933415525 podStartE2EDuration="1.933415525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:48.929755439 +0000 UTC m=+2422.870303346" watchObservedRunningTime="2026-10-29 23:22:48.933415525 +0000 UTC m=+2422.873963435" Oct 29 23:22:49.654315 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:49.654280 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:22:53.889612 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:22:53.889560 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:22:54.530669 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:54.530635 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:22:54.544140 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:54.544108 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:22:54.978877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:54.978848 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:22:57.656019 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:57.655983 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:22:57.883869 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:57.883839 4821 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:22:58.547842 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:22:58.547813 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:23:21.026140 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:21.026099 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k"] Oct 29 23:23:23.887490 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:23:23.887449 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:23:36.076624 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:36.076437 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" podUID="57f0a918-d7f5-4169-b2a9-106edf115d5e" containerName="main" containerID="cri-o://1e579c6a4836ff346b5445091f7302701a5a73673e436fe3a3ca85475d46662b" gracePeriod=45 Oct 29 23:23:36.556095 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:36.556057 4821 generic.go:356] "Generic (PLEG): container finished" podID="57f0a918-d7f5-4169-b2a9-106edf115d5e" containerID="1e579c6a4836ff346b5445091f7302701a5a73673e436fe3a3ca85475d46662b" exitCode=0 Oct 29 23:23:36.556383 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:36.556355 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" event={"ID":"57f0a918-d7f5-4169-b2a9-106edf115d5e","Type":"ContainerDied","Data":"1e579c6a4836ff346b5445091f7302701a5a73673e436fe3a3ca85475d46662b"} Oct 29 23:23:36.556587 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:36.556567 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" event={"ID":"57f0a918-d7f5-4169-b2a9-106edf115d5e","Type":"ContainerDied","Data":"e3538bc95cf89d01b161c5f426c81f8b250ab3f6045a2261b0d05e7445570bc2"} Oct 29 23:23:36.556721 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:36.556683 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3538bc95cf89d01b161c5f426c81f8b250ab3f6045a2261b0d05e7445570bc2" Oct 29 23:23:36.583437 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:36.583406 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:23:36.751025 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:36.750927 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57f0a918-d7f5-4169-b2a9-106edf115d5e-tls-certs\" (UniqueName: \"kubernetes.io/secret/57f0a918-d7f5-4169-b2a9-106edf115d5e-tls-certs\") pod \"57f0a918-d7f5-4169-b2a9-106edf115d5e\" (UID: \"57f0a918-d7f5-4169-b2a9-106edf115d5e\") " Oct 29 23:23:36.751025 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:36.750973 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f0a918-d7f5-4169-b2a9-106edf115d5e-kube-api-access-bpnv9\" (UniqueName: \"kubernetes.io/projected/57f0a918-d7f5-4169-b2a9-106edf115d5e-kube-api-access-bpnv9\") pod \"57f0a918-d7f5-4169-b2a9-106edf115d5e\" (UID: \"57f0a918-d7f5-4169-b2a9-106edf115d5e\") " Oct 29 23:23:36.751025 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:36.751009 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-tmp-dir\") pod \"57f0a918-d7f5-4169-b2a9-106edf115d5e\" (UID: \"57f0a918-d7f5-4169-b2a9-106edf115d5e\") " Oct 29 23:23:36.751420 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:36.751291 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-home\" (UniqueName: \"kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-home\") pod \"57f0a918-d7f5-4169-b2a9-106edf115d5e\" (UID: \"57f0a918-d7f5-4169-b2a9-106edf115d5e\") " Oct 29 23:23:36.751493 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:36.751403 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-tmp-dir" pod "57f0a918-d7f5-4169-b2a9-106edf115d5e" (UID: "57f0a918-d7f5-4169-b2a9-106edf115d5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:23:36.751585 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:36.751562 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-home" pod "57f0a918-d7f5-4169-b2a9-106edf115d5e" (UID: "57f0a918-d7f5-4169-b2a9-106edf115d5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:23:36.751661 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:36.751637 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-dshm\") pod \"57f0a918-d7f5-4169-b2a9-106edf115d5e\" (UID: \"57f0a918-d7f5-4169-b2a9-106edf115d5e\") " Oct 29 23:23:36.751992 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:36.751970 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-model-cache\") pod \"57f0a918-d7f5-4169-b2a9-106edf115d5e\" (UID: \"57f0a918-d7f5-4169-b2a9-106edf115d5e\") " Oct 29 23:23:36.752478 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:36.752383 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-model-cache" pod "57f0a918-d7f5-4169-b2a9-106edf115d5e" (UID: "57f0a918-d7f5-4169-b2a9-106edf115d5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:23:36.752972 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:36.752944 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:23:36.753081 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:36.752980 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:23:36.753081 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:36.752995 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:23:36.753746 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:36.753709 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57f0a918-d7f5-4169-b2a9-106edf115d5e-tls-certs" pod "57f0a918-d7f5-4169-b2a9-106edf115d5e" (UID: "57f0a918-d7f5-4169-b2a9-106edf115d5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:23:36.753845 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:36.753784 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f0a918-d7f5-4169-b2a9-106edf115d5e-kube-api-access-bpnv9" pod "57f0a918-d7f5-4169-b2a9-106edf115d5e" (UID: "57f0a918-d7f5-4169-b2a9-106edf115d5e"). InnerVolumeSpecName "kube-api-access-bpnv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:23:36.754007 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:36.753984 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-dshm" pod "57f0a918-d7f5-4169-b2a9-106edf115d5e" (UID: "57f0a918-d7f5-4169-b2a9-106edf115d5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:23:36.855169 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:36.855132 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57f0a918-d7f5-4169-b2a9-106edf115d5e-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:23:36.855169 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:36.855164 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpnv9\" (UniqueName: \"kubernetes.io/projected/57f0a918-d7f5-4169-b2a9-106edf115d5e-kube-api-access-bpnv9\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:23:36.855169 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:36.855173 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57f0a918-d7f5-4169-b2a9-106edf115d5e-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:23:37.566323 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:37.566245 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k" Oct 29 23:23:37.624487 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:37.624451 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k"] Oct 29 23:23:37.628969 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:37.628944 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-462fb2/test-model-462fb2-kserve-7b9c4d675-vn84k"] Oct 29 23:23:38.172100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:23:38.172069 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57f0a918-d7f5-4169-b2a9-106edf115d5e" path="/var/lib/kubelet/pods/57f0a918-d7f5-4169-b2a9-106edf115d5e/volumes" Oct 29 23:23:53.890919 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:23:53.890881 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:24:23.890674 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:24:23.890638 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:24:36.186894 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:36.186858 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98"] Oct 29 23:24:36.188106 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:36.187122 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98" podUID="e9c30d41-405c-49ee-931c-66f58edcd0dd" containerName="maas-api" containerID="cri-o://b9826eedda2fe7a734d00923ced1cbf70e355ad18bf7037c45e5292c8e59f372" gracePeriod=30 Oct 29 23:24:36.984224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:36.976160 4821 generic.go:356] "Generic (PLEG): container finished" podID="e9c30d41-405c-49ee-931c-66f58edcd0dd" containerID="b9826eedda2fe7a734d00923ced1cbf70e355ad18bf7037c45e5292c8e59f372" exitCode=0 Oct 29 23:24:36.984224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:36.976541 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98" event={"ID":"e9c30d41-405c-49ee-931c-66f58edcd0dd","Type":"ContainerDied","Data":"b9826eedda2fe7a734d00923ced1cbf70e355ad18bf7037c45e5292c8e59f372"} Oct 29 23:24:37.108228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:37.103731 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98" Oct 29 23:24:37.290800 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:37.290698 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9c30d41-405c-49ee-931c-66f58edcd0dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c30d41-405c-49ee-931c-66f58edcd0dd-metrics-certs\") pod \"e9c30d41-405c-49ee-931c-66f58edcd0dd\" (UID: \"e9c30d41-405c-49ee-931c-66f58edcd0dd\") " Oct 29 23:24:37.291338 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:37.290917 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c30d41-405c-49ee-931c-66f58edcd0dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9c30d41-405c-49ee-931c-66f58edcd0dd-openshift-service-ca\") pod \"e9c30d41-405c-49ee-931c-66f58edcd0dd\" (UID: \"e9c30d41-405c-49ee-931c-66f58edcd0dd\") " Oct 29 23:24:37.291338 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:37.290982 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c30d41-405c-49ee-931c-66f58edcd0dd-kube-api-access-cbxz2\" (UniqueName: \"kubernetes.io/projected/e9c30d41-405c-49ee-931c-66f58edcd0dd-kube-api-access-cbxz2\") pod \"e9c30d41-405c-49ee-931c-66f58edcd0dd\" (UID: \"e9c30d41-405c-49ee-931c-66f58edcd0dd\") " Oct 29 23:24:37.291338 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:37.291112 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9c30d41-405c-49ee-931c-66f58edcd0dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9c30d41-405c-49ee-931c-66f58edcd0dd-maas-api-tls\") pod \"e9c30d41-405c-49ee-931c-66f58edcd0dd\" (UID: \"e9c30d41-405c-49ee-931c-66f58edcd0dd\") " Oct 29 23:24:37.295280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:37.295252 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c30d41-405c-49ee-931c-66f58edcd0dd-openshift-service-ca" pod "e9c30d41-405c-49ee-931c-66f58edcd0dd" (UID: "e9c30d41-405c-49ee-931c-66f58edcd0dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:24:37.295951 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:37.295914 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c30d41-405c-49ee-931c-66f58edcd0dd-metrics-certs" pod "e9c30d41-405c-49ee-931c-66f58edcd0dd" (UID: "e9c30d41-405c-49ee-931c-66f58edcd0dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:24:37.297665 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:37.297644 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c30d41-405c-49ee-931c-66f58edcd0dd-maas-api-tls" pod "e9c30d41-405c-49ee-931c-66f58edcd0dd" (UID: "e9c30d41-405c-49ee-931c-66f58edcd0dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:24:37.297861 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:37.297833 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c30d41-405c-49ee-931c-66f58edcd0dd-kube-api-access-cbxz2" pod "e9c30d41-405c-49ee-931c-66f58edcd0dd" (UID: "e9c30d41-405c-49ee-931c-66f58edcd0dd"). InnerVolumeSpecName "kube-api-access-cbxz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:24:37.394112 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:37.394078 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbxz2\" (UniqueName: \"kubernetes.io/projected/e9c30d41-405c-49ee-931c-66f58edcd0dd-kube-api-access-cbxz2\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:24:37.394112 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:37.394110 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9c30d41-405c-49ee-931c-66f58edcd0dd-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:24:37.394112 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:37.394121 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9c30d41-405c-49ee-931c-66f58edcd0dd-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:24:37.394382 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:37.394130 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9c30d41-405c-49ee-931c-66f58edcd0dd-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:24:37.989138 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:37.989103 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98" Oct 29 23:24:37.989364 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:37.989113 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98" event={"ID":"e9c30d41-405c-49ee-931c-66f58edcd0dd","Type":"ContainerDied","Data":"49f461df9f270beb3a204a55110d8fae5dcb6940c30a3396d3e3fd9a394d04fb"} Oct 29 23:24:37.989364 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:37.989298 4821 scope.go:122] "RemoveContainer" containerID="b9826eedda2fe7a734d00923ced1cbf70e355ad18bf7037c45e5292c8e59f372" Oct 29 23:24:38.049136 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:38.049108 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98"] Oct 29 23:24:38.061334 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:38.061307 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9e7cc9-7d9c758d7b-vrc98"] Oct 29 23:24:38.172758 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:38.172726 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c30d41-405c-49ee-931c-66f58edcd0dd" path="/var/lib/kubelet/pods/e9c30d41-405c-49ee-931c-66f58edcd0dd/volumes" Oct 29 23:24:41.335373 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:41.335337 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8"] Oct 29 23:24:41.377341 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:41.377306 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8"] Oct 29 23:24:46.336697 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:46.336634 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8" podUID="60b4edf6-47dc-4829-bdaf-78c062cdce78" containerName="payload-processing" containerID="cri-o://ae00281d351a4ae41ec596da95b36eaa3ad4cb063f858a6a9231d0608296e816" gracePeriod=25 Oct 29 23:24:46.379377 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:46.379322 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8" podUID="ddb7ef9a-3948-4f03-ab50-3a9629a3c67d" containerName="payload-pre-processing" containerID="cri-o://f400b0bc8b7a4e887ba09dd4026101407040fa498d529e4e259b2c70384312cf" gracePeriod=25 Oct 29 23:24:53.889360 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:24:53.889322 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:24:55.426529 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:55.426489 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8" podUID="ddb7ef9a-3948-4f03-ab50-3a9629a3c67d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:24:55.426985 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:55.426499 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8" podUID="60b4edf6-47dc-4829-bdaf-78c062cdce78" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:24:56.812254 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:24:56.812192 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj"] Oct 29 23:25:05.426462 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:05.426409 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8" podUID="60b4edf6-47dc-4829-bdaf-78c062cdce78" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:25:05.426890 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:05.426409 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8" podUID="ddb7ef9a-3948-4f03-ab50-3a9629a3c67d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:25:06.681720 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:06.681691 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8" Oct 29 23:25:06.685588 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:06.685566 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8" Oct 29 23:25:06.855476 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:06.855435 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60b4edf6-47dc-4829-bdaf-78c062cdce78-kube-api-access-6fz6m\" (UniqueName: \"kubernetes.io/projected/60b4edf6-47dc-4829-bdaf-78c062cdce78-kube-api-access-6fz6m\") pod \"60b4edf6-47dc-4829-bdaf-78c062cdce78\" (UID: \"60b4edf6-47dc-4829-bdaf-78c062cdce78\") " Oct 29 23:25:06.855656 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:06.855541 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60b4edf6-47dc-4829-bdaf-78c062cdce78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b4edf6-47dc-4829-bdaf-78c062cdce78-plugins-config-volume\") pod \"60b4edf6-47dc-4829-bdaf-78c062cdce78\" (UID: \"60b4edf6-47dc-4829-bdaf-78c062cdce78\") " Oct 29 23:25:06.855656 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:06.855588 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddb7ef9a-3948-4f03-ab50-3a9629a3c67d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb7ef9a-3948-4f03-ab50-3a9629a3c67d-plugins-config-volume\") pod \"ddb7ef9a-3948-4f03-ab50-3a9629a3c67d\" (UID: \"ddb7ef9a-3948-4f03-ab50-3a9629a3c67d\") " Oct 29 23:25:06.855990 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:06.855928 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb7ef9a-3948-4f03-ab50-3a9629a3c67d-kube-api-access-f9hkb\" (UniqueName: \"kubernetes.io/projected/ddb7ef9a-3948-4f03-ab50-3a9629a3c67d-kube-api-access-f9hkb\") pod \"ddb7ef9a-3948-4f03-ab50-3a9629a3c67d\" (UID: \"ddb7ef9a-3948-4f03-ab50-3a9629a3c67d\") " Oct 29 23:25:06.856112 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:06.856043 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b4edf6-47dc-4829-bdaf-78c062cdce78-plugins-config-volume" pod "60b4edf6-47dc-4829-bdaf-78c062cdce78" (UID: "60b4edf6-47dc-4829-bdaf-78c062cdce78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:06.856401 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:06.856373 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddb7ef9a-3948-4f03-ab50-3a9629a3c67d-plugins-config-volume" pod "ddb7ef9a-3948-4f03-ab50-3a9629a3c67d" (UID: "ddb7ef9a-3948-4f03-ab50-3a9629a3c67d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:06.857083 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:06.857056 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b4edf6-47dc-4829-bdaf-78c062cdce78-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:06.857280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:06.857088 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb7ef9a-3948-4f03-ab50-3a9629a3c67d-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:06.858328 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:06.858305 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb7ef9a-3948-4f03-ab50-3a9629a3c67d-kube-api-access-f9hkb" pod "ddb7ef9a-3948-4f03-ab50-3a9629a3c67d" (UID: "ddb7ef9a-3948-4f03-ab50-3a9629a3c67d"). InnerVolumeSpecName "kube-api-access-f9hkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:06.858416 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:06.858396 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b4edf6-47dc-4829-bdaf-78c062cdce78-kube-api-access-6fz6m" pod "60b4edf6-47dc-4829-bdaf-78c062cdce78" (UID: "60b4edf6-47dc-4829-bdaf-78c062cdce78"). InnerVolumeSpecName "kube-api-access-6fz6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:06.959049 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:06.958957 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9hkb\" (UniqueName: \"kubernetes.io/projected/ddb7ef9a-3948-4f03-ab50-3a9629a3c67d-kube-api-access-f9hkb\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:06.959049 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:06.958989 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fz6m\" (UniqueName: \"kubernetes.io/projected/60b4edf6-47dc-4829-bdaf-78c062cdce78-kube-api-access-6fz6m\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:07.322658 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:07.322557 4821 generic.go:356] "Generic (PLEG): container finished" podID="60b4edf6-47dc-4829-bdaf-78c062cdce78" containerID="ae00281d351a4ae41ec596da95b36eaa3ad4cb063f858a6a9231d0608296e816" exitCode=0 Oct 29 23:25:07.322658 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:07.322631 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8" Oct 29 23:25:07.323148 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:07.322720 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8" event={"ID":"60b4edf6-47dc-4829-bdaf-78c062cdce78","Type":"ContainerDied","Data":"ae00281d351a4ae41ec596da95b36eaa3ad4cb063f858a6a9231d0608296e816"} Oct 29 23:25:07.323318 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:07.323288 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8" event={"ID":"60b4edf6-47dc-4829-bdaf-78c062cdce78","Type":"ContainerDied","Data":"e4bd78eb169a6da1f9ede589f763812282103157b83a5ac2adc60c77cbaecc28"} Oct 29 23:25:07.323418 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:07.323319 4821 scope.go:122] "RemoveContainer" containerID="ae00281d351a4ae41ec596da95b36eaa3ad4cb063f858a6a9231d0608296e816" Oct 29 23:25:07.330936 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:07.330898 4821 generic.go:356] "Generic (PLEG): container finished" podID="ddb7ef9a-3948-4f03-ab50-3a9629a3c67d" containerID="f400b0bc8b7a4e887ba09dd4026101407040fa498d529e4e259b2c70384312cf" exitCode=0 Oct 29 23:25:07.331065 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:07.330956 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8" Oct 29 23:25:07.331065 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:07.330980 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8" event={"ID":"ddb7ef9a-3948-4f03-ab50-3a9629a3c67d","Type":"ContainerDied","Data":"f400b0bc8b7a4e887ba09dd4026101407040fa498d529e4e259b2c70384312cf"} Oct 29 23:25:07.331065 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:07.331036 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8" event={"ID":"ddb7ef9a-3948-4f03-ab50-3a9629a3c67d","Type":"ContainerDied","Data":"3133e2a438319d7b0fd07dbfce8f4711dc3642fab239b2c3451721ea47f475e2"} Oct 29 23:25:07.337505 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:07.337486 4821 scope.go:122] "RemoveContainer" containerID="ae00281d351a4ae41ec596da95b36eaa3ad4cb063f858a6a9231d0608296e816" Oct 29 23:25:07.337808 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:25:07.337781 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae00281d351a4ae41ec596da95b36eaa3ad4cb063f858a6a9231d0608296e816\": container with ID starting with ae00281d351a4ae41ec596da95b36eaa3ad4cb063f858a6a9231d0608296e816 not found: ID does not exist" containerID="ae00281d351a4ae41ec596da95b36eaa3ad4cb063f858a6a9231d0608296e816" Oct 29 23:25:07.337892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:07.337817 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae00281d351a4ae41ec596da95b36eaa3ad4cb063f858a6a9231d0608296e816"} err="failed to get container status \"ae00281d351a4ae41ec596da95b36eaa3ad4cb063f858a6a9231d0608296e816\": rpc error: code = NotFound desc = could not find container \"ae00281d351a4ae41ec596da95b36eaa3ad4cb063f858a6a9231d0608296e816\": container with ID starting with ae00281d351a4ae41ec596da95b36eaa3ad4cb063f858a6a9231d0608296e816 not found: ID does not exist" Oct 29 23:25:07.337892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:07.337836 4821 scope.go:122] "RemoveContainer" containerID="f400b0bc8b7a4e887ba09dd4026101407040fa498d529e4e259b2c70384312cf" Oct 29 23:25:07.349389 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:07.349373 4821 scope.go:122] "RemoveContainer" containerID="f400b0bc8b7a4e887ba09dd4026101407040fa498d529e4e259b2c70384312cf" Oct 29 23:25:07.349672 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:25:07.349652 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f400b0bc8b7a4e887ba09dd4026101407040fa498d529e4e259b2c70384312cf\": container with ID starting with f400b0bc8b7a4e887ba09dd4026101407040fa498d529e4e259b2c70384312cf not found: ID does not exist" containerID="f400b0bc8b7a4e887ba09dd4026101407040fa498d529e4e259b2c70384312cf" Oct 29 23:25:07.349733 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:07.349679 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f400b0bc8b7a4e887ba09dd4026101407040fa498d529e4e259b2c70384312cf"} err="failed to get container status \"f400b0bc8b7a4e887ba09dd4026101407040fa498d529e4e259b2c70384312cf\": rpc error: code = NotFound desc = could not find container \"f400b0bc8b7a4e887ba09dd4026101407040fa498d529e4e259b2c70384312cf\": container with ID starting with f400b0bc8b7a4e887ba09dd4026101407040fa498d529e4e259b2c70384312cf not found: ID does not exist" Oct 29 23:25:07.383885 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:07.383858 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8"] Oct 29 23:25:07.397326 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:07.397285 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f4ce70-f47f9d564-hnjm8"] Oct 29 23:25:07.423305 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:07.423269 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8"] Oct 29 23:25:07.427767 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:07.427740 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f4ce70-64b5df955f-xq2n8"] Oct 29 23:25:08.173517 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:08.173487 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60b4edf6-47dc-4829-bdaf-78c062cdce78" path="/var/lib/kubelet/pods/60b4edf6-47dc-4829-bdaf-78c062cdce78/volumes" Oct 29 23:25:08.174026 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:08.174007 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddb7ef9a-3948-4f03-ab50-3a9629a3c67d" path="/var/lib/kubelet/pods/ddb7ef9a-3948-4f03-ab50-3a9629a3c67d/volumes" Oct 29 23:25:11.533073 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:11.533020 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj"] Oct 29 23:25:11.533647 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:11.533301 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" podUID="b26786c8-6181-41f2-a978-49b2123a690c" containerName="istio-proxy" containerID="cri-o://f1eac2fd1b7c7c546f44f512c1bcd1d8c04780e3cf1f66c3b04e5f2a3db55990" gracePeriod=30 Oct 29 23:25:11.749227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:11.736825 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj"] Oct 29 23:25:11.749227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:11.737058 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" podUID="c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" containerName="istio-proxy" containerID="cri-o://7068df05ddf4843625ed45bbecfe2e575c0ccacc92d93fec4bea2af66e2d2276" gracePeriod=30 Oct 29 23:25:11.906796 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:11.906704 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" podUID="631ed840-42fc-45b1-92a3-b05f747bd17f" containerName="main" containerID="cri-o://2a493bf7019ae63fcff79e450733944d8c895258bd464cfbee7271e7b17748ff" gracePeriod=45 Oct 29 23:25:12.224862 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.224834 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:25:12.313067 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.313031 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-tmp-dir\") pod \"631ed840-42fc-45b1-92a3-b05f747bd17f\" (UID: \"631ed840-42fc-45b1-92a3-b05f747bd17f\") " Oct 29 23:25:12.313271 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.313128 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/631ed840-42fc-45b1-92a3-b05f747bd17f-tls-certs\" (UniqueName: \"kubernetes.io/secret/631ed840-42fc-45b1-92a3-b05f747bd17f-tls-certs\") pod \"631ed840-42fc-45b1-92a3-b05f747bd17f\" (UID: \"631ed840-42fc-45b1-92a3-b05f747bd17f\") " Oct 29 23:25:12.313271 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.313172 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-home\" (UniqueName: \"kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-home\") pod \"631ed840-42fc-45b1-92a3-b05f747bd17f\" (UID: \"631ed840-42fc-45b1-92a3-b05f747bd17f\") " Oct 29 23:25:12.313416 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.313271 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-dshm\") pod \"631ed840-42fc-45b1-92a3-b05f747bd17f\" (UID: \"631ed840-42fc-45b1-92a3-b05f747bd17f\") " Oct 29 23:25:12.313416 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.313373 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-tmp-dir" pod "631ed840-42fc-45b1-92a3-b05f747bd17f" (UID: "631ed840-42fc-45b1-92a3-b05f747bd17f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:12.313537 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.313433 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-model-cache\") pod \"631ed840-42fc-45b1-92a3-b05f747bd17f\" (UID: \"631ed840-42fc-45b1-92a3-b05f747bd17f\") " Oct 29 23:25:12.313606 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.313534 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-home" pod "631ed840-42fc-45b1-92a3-b05f747bd17f" (UID: "631ed840-42fc-45b1-92a3-b05f747bd17f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:12.313606 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.313551 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/631ed840-42fc-45b1-92a3-b05f747bd17f-kube-api-access-k8p78\" (UniqueName: \"kubernetes.io/projected/631ed840-42fc-45b1-92a3-b05f747bd17f-kube-api-access-k8p78\") pod \"631ed840-42fc-45b1-92a3-b05f747bd17f\" (UID: \"631ed840-42fc-45b1-92a3-b05f747bd17f\") " Oct 29 23:25:12.313725 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.313692 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-model-cache" pod "631ed840-42fc-45b1-92a3-b05f747bd17f" (UID: "631ed840-42fc-45b1-92a3-b05f747bd17f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:12.314507 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.314478 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:12.314507 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.314508 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:12.314717 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.314524 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:12.316362 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.316320 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-dshm" pod "631ed840-42fc-45b1-92a3-b05f747bd17f" (UID: "631ed840-42fc-45b1-92a3-b05f747bd17f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:12.316507 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.316392 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631ed840-42fc-45b1-92a3-b05f747bd17f-kube-api-access-k8p78" pod "631ed840-42fc-45b1-92a3-b05f747bd17f" (UID: "631ed840-42fc-45b1-92a3-b05f747bd17f"). InnerVolumeSpecName "kube-api-access-k8p78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:12.317720 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.317682 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/631ed840-42fc-45b1-92a3-b05f747bd17f-tls-certs" pod "631ed840-42fc-45b1-92a3-b05f747bd17f" (UID: "631ed840-42fc-45b1-92a3-b05f747bd17f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:25:12.415824 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.415781 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/631ed840-42fc-45b1-92a3-b05f747bd17f-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:12.415824 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.415816 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/631ed840-42fc-45b1-92a3-b05f747bd17f-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:12.415824 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.415829 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8p78\" (UniqueName: \"kubernetes.io/projected/631ed840-42fc-45b1-92a3-b05f747bd17f-kube-api-access-k8p78\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:12.435239 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.435186 4821 generic.go:356] "Generic (PLEG): container finished" podID="631ed840-42fc-45b1-92a3-b05f747bd17f" containerID="2a493bf7019ae63fcff79e450733944d8c895258bd464cfbee7271e7b17748ff" exitCode=0 Oct 29 23:25:12.435425 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.435273 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" Oct 29 23:25:12.435425 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.435271 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" event={"ID":"631ed840-42fc-45b1-92a3-b05f747bd17f","Type":"ContainerDied","Data":"2a493bf7019ae63fcff79e450733944d8c895258bd464cfbee7271e7b17748ff"} Oct 29 23:25:12.435425 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.435330 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj" event={"ID":"631ed840-42fc-45b1-92a3-b05f747bd17f","Type":"ContainerDied","Data":"7f515e315a3a110fc3d2c898f90b6c9f973f5207ed2322d99c703d488c576451"} Oct 29 23:25:12.435425 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.435353 4821 scope.go:122] "RemoveContainer" containerID="2a493bf7019ae63fcff79e450733944d8c895258bd464cfbee7271e7b17748ff" Oct 29 23:25:12.448967 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.448947 4821 scope.go:122] "RemoveContainer" containerID="2a493bf7019ae63fcff79e450733944d8c895258bd464cfbee7271e7b17748ff" Oct 29 23:25:12.449302 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:25:12.449269 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a493bf7019ae63fcff79e450733944d8c895258bd464cfbee7271e7b17748ff\": container with ID starting with 2a493bf7019ae63fcff79e450733944d8c895258bd464cfbee7271e7b17748ff not found: ID does not exist" containerID="2a493bf7019ae63fcff79e450733944d8c895258bd464cfbee7271e7b17748ff" Oct 29 23:25:12.449379 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.449296 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a493bf7019ae63fcff79e450733944d8c895258bd464cfbee7271e7b17748ff"} err="failed to get container status \"2a493bf7019ae63fcff79e450733944d8c895258bd464cfbee7271e7b17748ff\": rpc error: code = NotFound desc = could not find container \"2a493bf7019ae63fcff79e450733944d8c895258bd464cfbee7271e7b17748ff\": container with ID starting with 2a493bf7019ae63fcff79e450733944d8c895258bd464cfbee7271e7b17748ff not found: ID does not exist" Oct 29 23:25:12.492623 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.492578 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj"] Oct 29 23:25:12.496906 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:12.496884 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-faa4fc/test-model-faa4fc-kserve-86f78b665c-ncfbj"] Oct 29 23:25:13.911966 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:13.911928 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:25:13.912432 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:13.911994 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" podUID="c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:25:14.173265 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:14.173174 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="631ed840-42fc-45b1-92a3-b05f747bd17f" path="/var/lib/kubelet/pods/631ed840-42fc-45b1-92a3-b05f747bd17f/volumes" Oct 29 23:25:17.412319 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.412283 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:25:17.446689 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.446643 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-workload-certs\") pod \"b26786c8-6181-41f2-a978-49b2123a690c\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " Oct 29 23:25:17.446836 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.446698 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b26786c8-6181-41f2-a978-49b2123a690c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b26786c8-6181-41f2-a978-49b2123a690c-istio-podinfo\") pod \"b26786c8-6181-41f2-a978-49b2123a690c\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " Oct 29 23:25:17.446891 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.446846 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b26786c8-6181-41f2-a978-49b2123a690c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b26786c8-6181-41f2-a978-49b2123a690c-istiod-ca-cert\") pod \"b26786c8-6181-41f2-a978-49b2123a690c\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " Oct 29 23:25:17.446891 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.446878 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-istio-data\") pod \"b26786c8-6181-41f2-a978-49b2123a690c\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " Oct 29 23:25:17.446998 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.446942 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-workload-certs" pod "b26786c8-6181-41f2-a978-49b2123a690c" (UID: "b26786c8-6181-41f2-a978-49b2123a690c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:17.447068 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.446999 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b26786c8-6181-41f2-a978-49b2123a690c-kube-api-access-lrtbs\" (UniqueName: \"kubernetes.io/projected/b26786c8-6181-41f2-a978-49b2123a690c-kube-api-access-lrtbs\") pod \"b26786c8-6181-41f2-a978-49b2123a690c\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " Oct 29 23:25:17.447068 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.447051 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b26786c8-6181-41f2-a978-49b2123a690c-istio-token\" (UniqueName: \"kubernetes.io/projected/b26786c8-6181-41f2-a978-49b2123a690c-istio-token\") pod \"b26786c8-6181-41f2-a978-49b2123a690c\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " Oct 29 23:25:17.447272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.447213 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-istio-data" pod "b26786c8-6181-41f2-a978-49b2123a690c" (UID: "b26786c8-6181-41f2-a978-49b2123a690c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:17.447272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.447245 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-credential-socket\") pod \"b26786c8-6181-41f2-a978-49b2123a690c\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " Oct 29 23:25:17.447434 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.447364 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-istio-envoy\") pod \"b26786c8-6181-41f2-a978-49b2123a690c\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " Oct 29 23:25:17.447511 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.447432 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-workload-socket\") pod \"b26786c8-6181-41f2-a978-49b2123a690c\" (UID: \"b26786c8-6181-41f2-a978-49b2123a690c\") " Oct 29 23:25:17.447832 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.447589 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-credential-socket" pod "b26786c8-6181-41f2-a978-49b2123a690c" (UID: "b26786c8-6181-41f2-a978-49b2123a690c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:17.447832 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.447772 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b26786c8-6181-41f2-a978-49b2123a690c-istiod-ca-cert" pod "b26786c8-6181-41f2-a978-49b2123a690c" (UID: "b26786c8-6181-41f2-a978-49b2123a690c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:17.448431 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.448009 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-workload-socket" pod "b26786c8-6181-41f2-a978-49b2123a690c" (UID: "b26786c8-6181-41f2-a978-49b2123a690c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:17.448431 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.448290 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.448431 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.448315 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b26786c8-6181-41f2-a978-49b2123a690c-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.448431 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.448328 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.448431 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.448341 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.448431 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.448356 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.449784 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.449758 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b26786c8-6181-41f2-a978-49b2123a690c-istio-podinfo" pod "b26786c8-6181-41f2-a978-49b2123a690c" (UID: "b26786c8-6181-41f2-a978-49b2123a690c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:25:17.450590 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.450559 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b26786c8-6181-41f2-a978-49b2123a690c-kube-api-access-lrtbs" pod "b26786c8-6181-41f2-a978-49b2123a690c" (UID: "b26786c8-6181-41f2-a978-49b2123a690c"). InnerVolumeSpecName "kube-api-access-lrtbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:17.451034 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.451007 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b26786c8-6181-41f2-a978-49b2123a690c-istio-token" pod "b26786c8-6181-41f2-a978-49b2123a690c" (UID: "b26786c8-6181-41f2-a978-49b2123a690c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:17.451283 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.451262 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-istio-envoy" pod "b26786c8-6181-41f2-a978-49b2123a690c" (UID: "b26786c8-6181-41f2-a978-49b2123a690c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:17.550340 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.550302 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrtbs\" (UniqueName: \"kubernetes.io/projected/b26786c8-6181-41f2-a978-49b2123a690c-kube-api-access-lrtbs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.550340 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.550340 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b26786c8-6181-41f2-a978-49b2123a690c-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.550592 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.550354 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b26786c8-6181-41f2-a978-49b2123a690c-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.550592 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.550366 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b26786c8-6181-41f2-a978-49b2123a690c-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.618311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.618281 4821 generic.go:356] "Generic (PLEG): container finished" podID="b26786c8-6181-41f2-a978-49b2123a690c" containerID="f1eac2fd1b7c7c546f44f512c1bcd1d8c04780e3cf1f66c3b04e5f2a3db55990" exitCode=0 Oct 29 23:25:17.618471 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.618400 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" event={"ID":"b26786c8-6181-41f2-a978-49b2123a690c","Type":"ContainerDied","Data":"f1eac2fd1b7c7c546f44f512c1bcd1d8c04780e3cf1f66c3b04e5f2a3db55990"} Oct 29 23:25:17.618553 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.618470 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" event={"ID":"b26786c8-6181-41f2-a978-49b2123a690c","Type":"ContainerDied","Data":"a9db1784f8a439904d4c7085743d8f619375c63ed7451bbc36a21a5db0e56e2e"} Oct 29 23:25:17.618553 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.618492 4821 scope.go:122] "RemoveContainer" containerID="f1eac2fd1b7c7c546f44f512c1bcd1d8c04780e3cf1f66c3b04e5f2a3db55990" Oct 29 23:25:17.618669 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.618601 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj" Oct 29 23:25:17.627635 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.627603 4821 generic.go:356] "Generic (PLEG): container finished" podID="c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" containerID="7068df05ddf4843625ed45bbecfe2e575c0ccacc92d93fec4bea2af66e2d2276" exitCode=0 Oct 29 23:25:17.627750 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.627680 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" event={"ID":"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a","Type":"ContainerDied","Data":"7068df05ddf4843625ed45bbecfe2e575c0ccacc92d93fec4bea2af66e2d2276"} Oct 29 23:25:17.673101 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.673079 4821 scope.go:122] "RemoveContainer" containerID="f1eac2fd1b7c7c546f44f512c1bcd1d8c04780e3cf1f66c3b04e5f2a3db55990" Oct 29 23:25:17.673420 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:25:17.673396 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1eac2fd1b7c7c546f44f512c1bcd1d8c04780e3cf1f66c3b04e5f2a3db55990\": container with ID starting with f1eac2fd1b7c7c546f44f512c1bcd1d8c04780e3cf1f66c3b04e5f2a3db55990 not found: ID does not exist" containerID="f1eac2fd1b7c7c546f44f512c1bcd1d8c04780e3cf1f66c3b04e5f2a3db55990" Oct 29 23:25:17.673496 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.673458 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1eac2fd1b7c7c546f44f512c1bcd1d8c04780e3cf1f66c3b04e5f2a3db55990"} err="failed to get container status \"f1eac2fd1b7c7c546f44f512c1bcd1d8c04780e3cf1f66c3b04e5f2a3db55990\": rpc error: code = NotFound desc = could not find container \"f1eac2fd1b7c7c546f44f512c1bcd1d8c04780e3cf1f66c3b04e5f2a3db55990\": container with ID starting with f1eac2fd1b7c7c546f44f512c1bcd1d8c04780e3cf1f66c3b04e5f2a3db55990 not found: ID does not exist" Oct 29 23:25:17.673496 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.673487 4821 scope.go:122] "RemoveContainer" containerID="d1217ac883fb396639158d9a394a35279bb12c6ba71cb0290eafd90e822f2101" Oct 29 23:25:17.678028 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.678008 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:25:17.714774 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.714742 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj"] Oct 29 23:25:17.720994 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.720964 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1056cc-openshift-default-7b4b9768c9-ww9nj"] Oct 29 23:25:17.753506 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.753481 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-workload-certs\") pod \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " Oct 29 23:25:17.753690 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.753612 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istiod-ca-cert\") pod \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " Oct 29 23:25:17.753761 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.753734 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-credential-socket\") pod \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " Oct 29 23:25:17.753761 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.753740 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-workload-certs" pod "c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" (UID: "c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:17.753868 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.753797 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-envoy\") pod \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " Oct 29 23:25:17.753930 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.753884 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-data\") pod \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " Oct 29 23:25:17.753984 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.753927 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-workload-socket\") pod \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " Oct 29 23:25:17.753984 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.753961 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-kube-api-access-5rbz9\" (UniqueName: \"kubernetes.io/projected/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-kube-api-access-5rbz9\") pod \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " Oct 29 23:25:17.754091 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.754004 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-credential-socket" pod "c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" (UID: "c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:17.754091 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.754031 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istiod-ca-cert" pod "c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" (UID: "c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:17.754091 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.754073 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-podinfo\") pod \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " Oct 29 23:25:17.754310 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.754232 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-data" pod "c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" (UID: "c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:17.754310 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.754284 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-token\" (UniqueName: \"kubernetes.io/projected/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-token\") pod \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\" (UID: \"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a\") " Oct 29 23:25:17.754443 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.754375 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-workload-socket" pod "c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" (UID: "c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:17.755505 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.755477 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.755631 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.755508 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.755631 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.755522 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.755631 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.755535 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.755631 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.755550 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.757032 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.757004 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-podinfo" pod "c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" (UID: "c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:25:17.757130 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.757113 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-kube-api-access-5rbz9" pod "c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" (UID: "c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a"). InnerVolumeSpecName "kube-api-access-5rbz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:17.757350 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.757324 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-token" pod "c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" (UID: "c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:17.757554 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.757534 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-envoy" pod "c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" (UID: "c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:17.856911 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.856872 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.856911 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.856904 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.856911 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.856913 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rbz9\" (UniqueName: \"kubernetes.io/projected/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-kube-api-access-5rbz9\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:17.857158 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:17.856923 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:18.172877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:18.172843 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b26786c8-6181-41f2-a978-49b2123a690c" path="/var/lib/kubelet/pods/b26786c8-6181-41f2-a978-49b2123a690c/volumes" Oct 29 23:25:18.637666 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:18.637634 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" Oct 29 23:25:18.637666 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:18.637658 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj" event={"ID":"c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a","Type":"ContainerDied","Data":"4b1ca03a34470000e79fb4c5e4942c83785d62c30599e716fe925b2eb1da75eb"} Oct 29 23:25:18.638307 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:18.637719 4821 scope.go:122] "RemoveContainer" containerID="7068df05ddf4843625ed45bbecfe2e575c0ccacc92d93fec4bea2af66e2d2276" Oct 29 23:25:18.763879 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:18.763840 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj"] Oct 29 23:25:18.768341 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:18.768315 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-67363b-openshift-default-745d499d87-dnsqj"] Oct 29 23:25:19.992840 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:19.992172 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq"] Oct 29 23:25:20.172582 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:20.172547 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" path="/var/lib/kubelet/pods/c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a/volumes" Oct 29 23:25:22.616194 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:22.616154 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk"] Oct 29 23:25:23.623463 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:23.623428 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s"] Oct 29 23:25:23.623934 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:23.623635 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" podUID="3b626717-62b8-428e-8371-9244e9f3e55f" containerName="istio-proxy" containerID="cri-o://5d7f9de66f2f49a52a155a04af7c19b03061860c1d58826c5a22722d011c2270" gracePeriod=30 Oct 29 23:25:23.887902 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:25:23.887803 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:25:24.992564 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:24.992521 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq" podUID="27c4bd01-02f4-4c07-8792-4f880626a8c0" containerName="payload-pre-processing" containerID="cri-o://d741cd0c92a2e9aefdc43d89d9a4b4908b37a9ecff36883fb2eab788a02d9595" gracePeriod=25 Oct 29 23:25:25.147427 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:25.147382 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw"] Oct 29 23:25:25.147674 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:25.147649 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw" podUID="41006044-c141-447c-8707-cf9d8d541c6b" containerName="maas-api" containerID="cri-o://40bd4fccbcdbfd3728b0c4882237fbf4f6404cdb9bacffc319e039de79048fa1" gracePeriod=30 Oct 29 23:25:25.916223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:25.906952 4821 generic.go:356] "Generic (PLEG): container finished" podID="41006044-c141-447c-8707-cf9d8d541c6b" containerID="40bd4fccbcdbfd3728b0c4882237fbf4f6404cdb9bacffc319e039de79048fa1" exitCode=0 Oct 29 23:25:25.916223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:25.909892 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw" event={"ID":"41006044-c141-447c-8707-cf9d8d541c6b","Type":"ContainerDied","Data":"40bd4fccbcdbfd3728b0c4882237fbf4f6404cdb9bacffc319e039de79048fa1"} Oct 29 23:25:26.027460 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:26.027425 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw" Oct 29 23:25:26.131923 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:26.131889 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41006044-c141-447c-8707-cf9d8d541c6b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41006044-c141-447c-8707-cf9d8d541c6b-maas-api-tls\") pod \"41006044-c141-447c-8707-cf9d8d541c6b\" (UID: \"41006044-c141-447c-8707-cf9d8d541c6b\") " Oct 29 23:25:26.132099 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:26.131945 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41006044-c141-447c-8707-cf9d8d541c6b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41006044-c141-447c-8707-cf9d8d541c6b-openshift-service-ca\") pod \"41006044-c141-447c-8707-cf9d8d541c6b\" (UID: \"41006044-c141-447c-8707-cf9d8d541c6b\") " Oct 29 23:25:26.132099 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:26.131972 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41006044-c141-447c-8707-cf9d8d541c6b-kube-api-access-749zx\" (UniqueName: \"kubernetes.io/projected/41006044-c141-447c-8707-cf9d8d541c6b-kube-api-access-749zx\") pod \"41006044-c141-447c-8707-cf9d8d541c6b\" (UID: \"41006044-c141-447c-8707-cf9d8d541c6b\") " Oct 29 23:25:26.132297 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:26.132278 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41006044-c141-447c-8707-cf9d8d541c6b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/41006044-c141-447c-8707-cf9d8d541c6b-metrics-certs\") pod \"41006044-c141-447c-8707-cf9d8d541c6b\" (UID: \"41006044-c141-447c-8707-cf9d8d541c6b\") " Oct 29 23:25:26.132371 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:26.132345 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41006044-c141-447c-8707-cf9d8d541c6b-openshift-service-ca" pod "41006044-c141-447c-8707-cf9d8d541c6b" (UID: "41006044-c141-447c-8707-cf9d8d541c6b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:26.133042 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:26.133019 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41006044-c141-447c-8707-cf9d8d541c6b-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:26.134326 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:26.134300 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41006044-c141-447c-8707-cf9d8d541c6b-maas-api-tls" pod "41006044-c141-447c-8707-cf9d8d541c6b" (UID: "41006044-c141-447c-8707-cf9d8d541c6b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:25:26.134430 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:26.134417 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41006044-c141-447c-8707-cf9d8d541c6b-metrics-certs" pod "41006044-c141-447c-8707-cf9d8d541c6b" (UID: "41006044-c141-447c-8707-cf9d8d541c6b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:25:26.134430 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:26.134422 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41006044-c141-447c-8707-cf9d8d541c6b-kube-api-access-749zx" pod "41006044-c141-447c-8707-cf9d8d541c6b" (UID: "41006044-c141-447c-8707-cf9d8d541c6b"). InnerVolumeSpecName "kube-api-access-749zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:26.243537 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:26.243500 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41006044-c141-447c-8707-cf9d8d541c6b-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:26.243837 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:26.243819 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-749zx\" (UniqueName: \"kubernetes.io/projected/41006044-c141-447c-8707-cf9d8d541c6b-kube-api-access-749zx\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:26.243955 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:26.243938 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41006044-c141-447c-8707-cf9d8d541c6b-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:26.921841 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:26.921798 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw" Oct 29 23:25:26.922055 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:26.921802 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw" event={"ID":"41006044-c141-447c-8707-cf9d8d541c6b","Type":"ContainerDied","Data":"7c8926c02aba1eea6d2803ce9039e4677bb9adbbd9373faa8992c310a4eed0a4"} Oct 29 23:25:26.922055 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:26.921936 4821 scope.go:122] "RemoveContainer" containerID="40bd4fccbcdbfd3728b0c4882237fbf4f6404cdb9bacffc319e039de79048fa1" Oct 29 23:25:27.001307 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:27.001253 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw"] Oct 29 23:25:27.002143 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:27.002115 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f4ce70-66b578946-wxghw"] Oct 29 23:25:27.616735 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:27.616692 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk" podUID="e52400e6-b2db-4dca-b23d-ad9f9a81bf44" containerName="payload-pre-processing" containerID="cri-o://0ed57a09dd98f3049981c1e8c26b799da8836aeb5e34be23f0d3dbaf98895f9d" gracePeriod=25 Oct 29 23:25:28.182720 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:28.182667 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41006044-c141-447c-8707-cf9d8d541c6b" path="/var/lib/kubelet/pods/41006044-c141-447c-8707-cf9d8d541c6b/volumes" Oct 29 23:25:28.184976 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:28.184942 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c"] Oct 29 23:25:28.185167 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:28.185145 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" podUID="d3c8fc4f-604c-4938-b9e1-5745ac6558c7" containerName="istio-proxy" containerID="cri-o://2c4d24aae0b2829c32abe6b5ba1c0b2488448ce49fbccb97e38e5d462dd0e2c6" gracePeriod=30 Oct 29 23:25:28.918831 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:28.918804 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:25:28.942476 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:28.942447 4821 generic.go:356] "Generic (PLEG): container finished" podID="3b626717-62b8-428e-8371-9244e9f3e55f" containerID="5d7f9de66f2f49a52a155a04af7c19b03061860c1d58826c5a22722d011c2270" exitCode=0 Oct 29 23:25:28.942637 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:28.942493 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" Oct 29 23:25:28.942637 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:28.942528 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" event={"ID":"3b626717-62b8-428e-8371-9244e9f3e55f","Type":"ContainerDied","Data":"5d7f9de66f2f49a52a155a04af7c19b03061860c1d58826c5a22722d011c2270"} Oct 29 23:25:28.942637 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:28.942594 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s" event={"ID":"3b626717-62b8-428e-8371-9244e9f3e55f","Type":"ContainerDied","Data":"699185642e75f92af35271176e80ca113ee90ae0fd77fcb31f3504d805f50eda"} Oct 29 23:25:28.942637 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:28.942625 4821 scope.go:122] "RemoveContainer" containerID="5d7f9de66f2f49a52a155a04af7c19b03061860c1d58826c5a22722d011c2270" Oct 29 23:25:28.955653 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:28.955632 4821 scope.go:122] "RemoveContainer" containerID="5d7f9de66f2f49a52a155a04af7c19b03061860c1d58826c5a22722d011c2270" Oct 29 23:25:28.955980 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:25:28.955959 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d7f9de66f2f49a52a155a04af7c19b03061860c1d58826c5a22722d011c2270\": container with ID starting with 5d7f9de66f2f49a52a155a04af7c19b03061860c1d58826c5a22722d011c2270 not found: ID does not exist" containerID="5d7f9de66f2f49a52a155a04af7c19b03061860c1d58826c5a22722d011c2270" Oct 29 23:25:28.956060 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:28.955989 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7f9de66f2f49a52a155a04af7c19b03061860c1d58826c5a22722d011c2270"} err="failed to get container status \"5d7f9de66f2f49a52a155a04af7c19b03061860c1d58826c5a22722d011c2270\": rpc error: code = NotFound desc = could not find container \"5d7f9de66f2f49a52a155a04af7c19b03061860c1d58826c5a22722d011c2270\": container with ID starting with 5d7f9de66f2f49a52a155a04af7c19b03061860c1d58826c5a22722d011c2270 not found: ID does not exist" Oct 29 23:25:29.004491 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.004457 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-credential-socket\") pod \"3b626717-62b8-428e-8371-9244e9f3e55f\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " Oct 29 23:25:29.004786 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.004757 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-istio-envoy\") pod \"3b626717-62b8-428e-8371-9244e9f3e55f\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " Oct 29 23:25:29.004925 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.004816 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b626717-62b8-428e-8371-9244e9f3e55f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b626717-62b8-428e-8371-9244e9f3e55f-istiod-ca-cert\") pod \"3b626717-62b8-428e-8371-9244e9f3e55f\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " Oct 29 23:25:29.005010 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.004991 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-workload-certs\") pod \"3b626717-62b8-428e-8371-9244e9f3e55f\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " Oct 29 23:25:29.005075 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.005059 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-workload-socket\") pod \"3b626717-62b8-428e-8371-9244e9f3e55f\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " Oct 29 23:25:29.005157 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.005102 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b626717-62b8-428e-8371-9244e9f3e55f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b626717-62b8-428e-8371-9244e9f3e55f-istio-podinfo\") pod \"3b626717-62b8-428e-8371-9244e9f3e55f\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " Oct 29 23:25:29.005157 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.005142 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b626717-62b8-428e-8371-9244e9f3e55f-istio-token\" (UniqueName: \"kubernetes.io/projected/3b626717-62b8-428e-8371-9244e9f3e55f-istio-token\") pod \"3b626717-62b8-428e-8371-9244e9f3e55f\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " Oct 29 23:25:29.005335 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.005253 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-istio-data\") pod \"3b626717-62b8-428e-8371-9244e9f3e55f\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " Oct 29 23:25:29.005396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.005341 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b626717-62b8-428e-8371-9244e9f3e55f-kube-api-access-7rtbg\" (UniqueName: \"kubernetes.io/projected/3b626717-62b8-428e-8371-9244e9f3e55f-kube-api-access-7rtbg\") pod \"3b626717-62b8-428e-8371-9244e9f3e55f\" (UID: \"3b626717-62b8-428e-8371-9244e9f3e55f\") " Oct 29 23:25:29.007231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.006994 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-workload-certs" pod "3b626717-62b8-428e-8371-9244e9f3e55f" (UID: "3b626717-62b8-428e-8371-9244e9f3e55f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:29.009262 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.007516 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b626717-62b8-428e-8371-9244e9f3e55f-istiod-ca-cert" pod "3b626717-62b8-428e-8371-9244e9f3e55f" (UID: "3b626717-62b8-428e-8371-9244e9f3e55f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:29.009262 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.007774 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-credential-socket" pod "3b626717-62b8-428e-8371-9244e9f3e55f" (UID: "3b626717-62b8-428e-8371-9244e9f3e55f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:29.009262 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.008089 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-workload-socket" pod "3b626717-62b8-428e-8371-9244e9f3e55f" (UID: "3b626717-62b8-428e-8371-9244e9f3e55f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:29.009262 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.008520 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-istio-data" pod "3b626717-62b8-428e-8371-9244e9f3e55f" (UID: "3b626717-62b8-428e-8371-9244e9f3e55f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:29.012246 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.011321 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b626717-62b8-428e-8371-9244e9f3e55f-istio-token" pod "3b626717-62b8-428e-8371-9244e9f3e55f" (UID: "3b626717-62b8-428e-8371-9244e9f3e55f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:29.012246 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.011378 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b626717-62b8-428e-8371-9244e9f3e55f-istio-podinfo" pod "3b626717-62b8-428e-8371-9244e9f3e55f" (UID: "3b626717-62b8-428e-8371-9244e9f3e55f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:25:29.012246 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.011407 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b626717-62b8-428e-8371-9244e9f3e55f-kube-api-access-7rtbg" pod "3b626717-62b8-428e-8371-9244e9f3e55f" (UID: "3b626717-62b8-428e-8371-9244e9f3e55f"). InnerVolumeSpecName "kube-api-access-7rtbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:29.012246 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.011563 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-istio-envoy" pod "3b626717-62b8-428e-8371-9244e9f3e55f" (UID: "3b626717-62b8-428e-8371-9244e9f3e55f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:29.107482 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.107449 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b626717-62b8-428e-8371-9244e9f3e55f-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:29.107482 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.107481 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:29.107723 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.107493 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rtbg\" (UniqueName: \"kubernetes.io/projected/3b626717-62b8-428e-8371-9244e9f3e55f-kube-api-access-7rtbg\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:29.107723 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.107509 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:29.107723 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.107522 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:29.107723 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.107536 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b626717-62b8-428e-8371-9244e9f3e55f-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:29.107723 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.107548 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:29.107723 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.107558 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b626717-62b8-428e-8371-9244e9f3e55f-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:29.107723 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.107569 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b626717-62b8-428e-8371-9244e9f3e55f-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:29.293700 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.293669 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s"] Oct 29 23:25:29.306656 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:29.306632 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b0c3c3-openshift-default-669db7bb59-frp8s"] Oct 29 23:25:30.006905 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:30.006870 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk" podUID="e52400e6-b2db-4dca-b23d-ad9f9a81bf44" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:25:30.172260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:30.172224 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b626717-62b8-428e-8371-9244e9f3e55f" path="/var/lib/kubelet/pods/3b626717-62b8-428e-8371-9244e9f3e55f/volumes" Oct 29 23:25:30.214694 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:30.214663 4821 patch_prober.go:29] interesting pod/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:25:30.214855 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:30.214738 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" podUID="d3c8fc4f-604c-4938-b9e1-5745ac6558c7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:25:30.585466 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:30.585432 4821 scope.go:122] "RemoveContainer" containerID="1e579c6a4836ff346b5445091f7302701a5a73673e436fe3a3ca85475d46662b" Oct 29 23:25:33.366569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.366534 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl"] Oct 29 23:25:33.367433 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.367387 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" podUID="c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" containerName="istio-proxy" containerID="cri-o://c2422e9076a205fb6c3711163100b56c37c8faca0b2d7d48f4ac613e44633e8c" gracePeriod=30 Oct 29 23:25:33.755548 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.755178 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 23:25:33.855751 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.855705 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-workload-socket\") pod \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " Oct 29 23:25:33.855933 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.855801 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-kube-api-access-9z9pf\" (UniqueName: \"kubernetes.io/projected/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-kube-api-access-9z9pf\") pod \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " Oct 29 23:25:33.855933 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.855834 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-podinfo\") pod \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " Oct 29 23:25:33.855933 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.855898 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-envoy\") pod \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " Oct 29 23:25:33.856131 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.855939 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istiod-ca-cert\") pod \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " Oct 29 23:25:33.856131 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.855984 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-workload-socket" pod "d3c8fc4f-604c-4938-b9e1-5745ac6558c7" (UID: "d3c8fc4f-604c-4938-b9e1-5745ac6558c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:33.856131 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.856004 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-token\" (UniqueName: \"kubernetes.io/projected/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-token\") pod \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " Oct 29 23:25:33.856131 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.856043 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-credential-socket\") pod \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " Oct 29 23:25:33.856131 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.856113 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-data\") pod \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " Oct 29 23:25:33.856444 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.856191 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-workload-certs\") pod \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\" (UID: \"d3c8fc4f-604c-4938-b9e1-5745ac6558c7\") " Oct 29 23:25:33.856444 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.856382 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istiod-ca-cert" pod "d3c8fc4f-604c-4938-b9e1-5745ac6558c7" (UID: "d3c8fc4f-604c-4938-b9e1-5745ac6558c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:33.856548 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.856495 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-data" pod "d3c8fc4f-604c-4938-b9e1-5745ac6558c7" (UID: "d3c8fc4f-604c-4938-b9e1-5745ac6558c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:33.856548 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.856521 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-credential-socket" pod "d3c8fc4f-604c-4938-b9e1-5745ac6558c7" (UID: "d3c8fc4f-604c-4938-b9e1-5745ac6558c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:33.856680 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.856661 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-workload-certs" pod "d3c8fc4f-604c-4938-b9e1-5745ac6558c7" (UID: "d3c8fc4f-604c-4938-b9e1-5745ac6558c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:33.857382 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.857358 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:33.857476 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.857386 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:33.857476 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.857402 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:33.857476 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.857417 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:33.857476 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.857430 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:33.858544 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.858516 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-kube-api-access-9z9pf" pod "d3c8fc4f-604c-4938-b9e1-5745ac6558c7" (UID: "d3c8fc4f-604c-4938-b9e1-5745ac6558c7"). InnerVolumeSpecName "kube-api-access-9z9pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:33.858544 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.858532 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-token" pod "d3c8fc4f-604c-4938-b9e1-5745ac6558c7" (UID: "d3c8fc4f-604c-4938-b9e1-5745ac6558c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:33.858785 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.858764 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-podinfo" pod "d3c8fc4f-604c-4938-b9e1-5745ac6558c7" (UID: "d3c8fc4f-604c-4938-b9e1-5745ac6558c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:25:33.858785 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.858775 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-envoy" pod "d3c8fc4f-604c-4938-b9e1-5745ac6558c7" (UID: "d3c8fc4f-604c-4938-b9e1-5745ac6558c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:33.959064 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.958986 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:33.959064 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.959022 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z9pf\" (UniqueName: \"kubernetes.io/projected/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-kube-api-access-9z9pf\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:33.959064 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.959037 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:33.959064 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.959050 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c8fc4f-604c-4938-b9e1-5745ac6558c7-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:33.998702 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.998663 4821 generic.go:356] "Generic (PLEG): container finished" podID="d3c8fc4f-604c-4938-b9e1-5745ac6558c7" containerID="2c4d24aae0b2829c32abe6b5ba1c0b2488448ce49fbccb97e38e5d462dd0e2c6" exitCode=0 Oct 29 23:25:33.998898 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.998719 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" Oct 29 23:25:33.998898 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.998761 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" event={"ID":"d3c8fc4f-604c-4938-b9e1-5745ac6558c7","Type":"ContainerDied","Data":"2c4d24aae0b2829c32abe6b5ba1c0b2488448ce49fbccb97e38e5d462dd0e2c6"} Oct 29 23:25:33.998898 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.998869 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c" event={"ID":"d3c8fc4f-604c-4938-b9e1-5745ac6558c7","Type":"ContainerDied","Data":"38627a23370f3f31a0de0883c561d36e08a8ebf85ceeab8ec683d0a9716afbe3"} Oct 29 23:25:33.999102 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:33.998910 4821 scope.go:122] "RemoveContainer" containerID="2c4d24aae0b2829c32abe6b5ba1c0b2488448ce49fbccb97e38e5d462dd0e2c6" Oct 29 23:25:34.012825 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:34.012801 4821 scope.go:122] "RemoveContainer" containerID="2c4d24aae0b2829c32abe6b5ba1c0b2488448ce49fbccb97e38e5d462dd0e2c6" Oct 29 23:25:34.013111 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:25:34.013093 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c4d24aae0b2829c32abe6b5ba1c0b2488448ce49fbccb97e38e5d462dd0e2c6\": container with ID starting with 2c4d24aae0b2829c32abe6b5ba1c0b2488448ce49fbccb97e38e5d462dd0e2c6 not found: ID does not exist" containerID="2c4d24aae0b2829c32abe6b5ba1c0b2488448ce49fbccb97e38e5d462dd0e2c6" Oct 29 23:25:34.013179 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:34.013119 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c4d24aae0b2829c32abe6b5ba1c0b2488448ce49fbccb97e38e5d462dd0e2c6"} err="failed to get container status \"2c4d24aae0b2829c32abe6b5ba1c0b2488448ce49fbccb97e38e5d462dd0e2c6\": rpc error: code = NotFound desc = could not find container \"2c4d24aae0b2829c32abe6b5ba1c0b2488448ce49fbccb97e38e5d462dd0e2c6\": container with ID starting with 2c4d24aae0b2829c32abe6b5ba1c0b2488448ce49fbccb97e38e5d462dd0e2c6 not found: ID does not exist" Oct 29 23:25:34.057126 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:34.057096 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c"] Oct 29 23:25:34.064291 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:34.064265 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-08e15d-openshift-default-d579fdc75-z8j4c"] Oct 29 23:25:34.172497 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:34.172464 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3c8fc4f-604c-4938-b9e1-5745ac6558c7" path="/var/lib/kubelet/pods/d3c8fc4f-604c-4938-b9e1-5745ac6558c7/volumes" Oct 29 23:25:34.600954 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:34.600919 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq" podUID="27c4bd01-02f4-4c07-8792-4f880626a8c0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:25:34.846970 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:34.846935 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw"] Oct 29 23:25:34.847153 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:34.847116 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" podUID="073c5be5-712e-4634-82a1-9f957244d5e1" containerName="istio-proxy" containerID="cri-o://be155f2bf566b59089ba78a18a5fb3dc25269005ce0063e5b783a15127bf8e6d" gracePeriod=30 Oct 29 23:25:35.792876 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:35.792839 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk"] Oct 29 23:25:35.793431 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:35.793077 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" podUID="30e8d47c-27fb-40d8-86fb-eea11610630b" containerName="istio-proxy" containerID="cri-o://511573a6f4f5a42b2a375ee02c1f383872f7cb22915a0563dd6ef36fef9b9b84" gracePeriod=30 Oct 29 23:25:36.372466 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:36.372429 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf"] Oct 29 23:25:36.372667 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:36.372635 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" podUID="0fd71b7c-e947-4db2-97fb-e16431ec4a37" containerName="maas-api" containerID="cri-o://2a5e11fbdef469caa659a31d5ca210a0f06a7256cf679ac7afbdde0cf83883a3" gracePeriod=30 Oct 29 23:25:36.968224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:36.966600 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" Oct 29 23:25:37.078037 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:37.077933 4821 generic.go:356] "Generic (PLEG): container finished" podID="0fd71b7c-e947-4db2-97fb-e16431ec4a37" containerID="2a5e11fbdef469caa659a31d5ca210a0f06a7256cf679ac7afbdde0cf83883a3" exitCode=0 Oct 29 23:25:37.078339 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:37.078284 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" event={"ID":"0fd71b7c-e947-4db2-97fb-e16431ec4a37","Type":"ContainerDied","Data":"2a5e11fbdef469caa659a31d5ca210a0f06a7256cf679ac7afbdde0cf83883a3"} Oct 29 23:25:37.078545 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:37.078502 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" event={"ID":"0fd71b7c-e947-4db2-97fb-e16431ec4a37","Type":"ContainerDied","Data":"f15a7bbd70ec8d1fe2a6dc2604a41df166465e545d60beaac859c8eb40d45829"} Oct 29 23:25:37.078722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:37.078662 4821 scope.go:122] "RemoveContainer" containerID="2a5e11fbdef469caa659a31d5ca210a0f06a7256cf679ac7afbdde0cf83883a3" Oct 29 23:25:37.079903 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:37.079852 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" Oct 29 23:25:37.082165 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:37.082138 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fd71b7c-e947-4db2-97fb-e16431ec4a37-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fd71b7c-e947-4db2-97fb-e16431ec4a37-maas-api-tls\") pod \"0fd71b7c-e947-4db2-97fb-e16431ec4a37\" (UID: \"0fd71b7c-e947-4db2-97fb-e16431ec4a37\") " Oct 29 23:25:37.082511 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:37.082492 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fd71b7c-e947-4db2-97fb-e16431ec4a37-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fd71b7c-e947-4db2-97fb-e16431ec4a37-metrics-certs\") pod \"0fd71b7c-e947-4db2-97fb-e16431ec4a37\" (UID: \"0fd71b7c-e947-4db2-97fb-e16431ec4a37\") " Oct 29 23:25:37.082721 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:37.082704 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd71b7c-e947-4db2-97fb-e16431ec4a37-kube-api-access-58s9m\" (UniqueName: \"kubernetes.io/projected/0fd71b7c-e947-4db2-97fb-e16431ec4a37-kube-api-access-58s9m\") pod \"0fd71b7c-e947-4db2-97fb-e16431ec4a37\" (UID: \"0fd71b7c-e947-4db2-97fb-e16431ec4a37\") " Oct 29 23:25:37.083017 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:37.082995 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fd71b7c-e947-4db2-97fb-e16431ec4a37-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd71b7c-e947-4db2-97fb-e16431ec4a37-openshift-service-ca\") pod \"0fd71b7c-e947-4db2-97fb-e16431ec4a37\" (UID: \"0fd71b7c-e947-4db2-97fb-e16431ec4a37\") " Oct 29 23:25:37.084246 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:37.084179 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fd71b7c-e947-4db2-97fb-e16431ec4a37-openshift-service-ca" pod "0fd71b7c-e947-4db2-97fb-e16431ec4a37" (UID: "0fd71b7c-e947-4db2-97fb-e16431ec4a37"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:37.101400 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:37.101357 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd71b7c-e947-4db2-97fb-e16431ec4a37-kube-api-access-58s9m" pod "0fd71b7c-e947-4db2-97fb-e16431ec4a37" (UID: "0fd71b7c-e947-4db2-97fb-e16431ec4a37"). InnerVolumeSpecName "kube-api-access-58s9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:37.117915 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:37.116970 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fd71b7c-e947-4db2-97fb-e16431ec4a37-metrics-certs" pod "0fd71b7c-e947-4db2-97fb-e16431ec4a37" (UID: "0fd71b7c-e947-4db2-97fb-e16431ec4a37"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:25:37.190546 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:37.190502 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd71b7c-e947-4db2-97fb-e16431ec4a37-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:37.192104 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:37.192078 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fd71b7c-e947-4db2-97fb-e16431ec4a37-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:37.192346 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:37.192319 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58s9m\" (UniqueName: \"kubernetes.io/projected/0fd71b7c-e947-4db2-97fb-e16431ec4a37-kube-api-access-58s9m\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:37.204135 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:37.203998 4821 scope.go:122] "RemoveContainer" containerID="2a5e11fbdef469caa659a31d5ca210a0f06a7256cf679ac7afbdde0cf83883a3" Oct 29 23:25:37.206397 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:37.206326 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fd71b7c-e947-4db2-97fb-e16431ec4a37-maas-api-tls" pod "0fd71b7c-e947-4db2-97fb-e16431ec4a37" (UID: "0fd71b7c-e947-4db2-97fb-e16431ec4a37"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:25:37.207679 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:25:37.207584 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a5e11fbdef469caa659a31d5ca210a0f06a7256cf679ac7afbdde0cf83883a3\": container with ID starting with 2a5e11fbdef469caa659a31d5ca210a0f06a7256cf679ac7afbdde0cf83883a3 not found: ID does not exist" containerID="2a5e11fbdef469caa659a31d5ca210a0f06a7256cf679ac7afbdde0cf83883a3" Oct 29 23:25:37.207679 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:37.207629 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a5e11fbdef469caa659a31d5ca210a0f06a7256cf679ac7afbdde0cf83883a3"} err="failed to get container status \"2a5e11fbdef469caa659a31d5ca210a0f06a7256cf679ac7afbdde0cf83883a3\": rpc error: code = NotFound desc = could not find container \"2a5e11fbdef469caa659a31d5ca210a0f06a7256cf679ac7afbdde0cf83883a3\": container with ID starting with 2a5e11fbdef469caa659a31d5ca210a0f06a7256cf679ac7afbdde0cf83883a3 not found: ID does not exist" Oct 29 23:25:37.294559 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:37.294535 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fd71b7c-e947-4db2-97fb-e16431ec4a37-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:37.444875 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:37.444833 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf"] Oct 29 23:25:37.450958 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:37.450781 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf"] Oct 29 23:25:38.110967 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.110924 4821 patch_prober.go:29] interesting pod/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:25:38.111445 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.110993 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" podUID="30e8d47c-27fb-40d8-86fb-eea11610630b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:25:38.173138 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.173094 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fd71b7c-e947-4db2-97fb-e16431ec4a37" path="/var/lib/kubelet/pods/0fd71b7c-e947-4db2-97fb-e16431ec4a37/volumes" Oct 29 23:25:38.618541 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.618509 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz"] Oct 29 23:25:38.618751 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.618713 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" podUID="da46e2ae-5388-4445-8351-bcc2204a9dcf" containerName="istio-proxy" containerID="cri-o://4fc80bbdf569b4c193d1b0fa400f79242cf0a642dc0edaad6ef227e83a9061e0" gracePeriod=30 Oct 29 23:25:38.667247 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.659518 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp"] Oct 29 23:25:38.667247 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.659779 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" podUID="df9f4a39-1043-449e-b5b3-b57c82b618b3" containerName="istio-proxy" containerID="cri-o://c4948e75f358f5cd5e3cbf70b52b02fa3804855bdb9c464547466161a0df51ef" gracePeriod=30 Oct 29 23:25:38.669789 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.669748 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:25:38.817704 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.817664 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-data\") pod \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " Oct 29 23:25:38.817930 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.817791 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-kube-api-access-rm77h\" (UniqueName: \"kubernetes.io/projected/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-kube-api-access-rm77h\") pod \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " Oct 29 23:25:38.818011 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.817933 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-workload-certs\") pod \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " Oct 29 23:25:38.818070 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.818004 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-podinfo\") pod \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " Oct 29 23:25:38.818126 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.818065 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-credential-socket\") pod \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " Oct 29 23:25:38.818251 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.818193 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-envoy\") pod \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " Oct 29 23:25:38.818324 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.818256 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-token\" (UniqueName: \"kubernetes.io/projected/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-token\") pod \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " Oct 29 23:25:38.818324 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.818289 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-workload-socket\") pod \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " Oct 29 23:25:38.818462 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.818342 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istiod-ca-cert\") pod \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\" (UID: \"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe\") " Oct 29 23:25:38.819558 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.819529 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istiod-ca-cert" pod "c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" (UID: "c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:38.819961 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.819887 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-data" pod "c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" (UID: "c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:38.823713 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.820780 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-workload-certs" pod "c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" (UID: "c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:38.823713 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.820918 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-workload-socket" pod "c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" (UID: "c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:38.823713 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.821429 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-credential-socket" pod "c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" (UID: "c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:38.829067 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.829040 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-podinfo" pod "c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" (UID: "c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:25:38.829167 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.829078 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-token" pod "c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" (UID: "c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:38.829293 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.829179 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-kube-api-access-rm77h" pod "c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" (UID: "c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe"). InnerVolumeSpecName "kube-api-access-rm77h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:38.830495 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.830473 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-envoy" pod "c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" (UID: "c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:38.921230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.921116 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:38.921230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.921147 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:38.921230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.921158 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:38.921230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.921168 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:38.921230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.921177 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:38.921230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.921211 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:38.921230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.921220 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:38.921230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.921231 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:38.921230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:38.921239 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm77h\" (UniqueName: \"kubernetes.io/projected/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe-kube-api-access-rm77h\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:39.096106 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:39.096072 4821 generic.go:356] "Generic (PLEG): container finished" podID="c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" containerID="c2422e9076a205fb6c3711163100b56c37c8faca0b2d7d48f4ac613e44633e8c" exitCode=0 Oct 29 23:25:39.096316 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:39.096126 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" Oct 29 23:25:39.096316 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:39.096157 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" event={"ID":"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe","Type":"ContainerDied","Data":"c2422e9076a205fb6c3711163100b56c37c8faca0b2d7d48f4ac613e44633e8c"} Oct 29 23:25:39.096316 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:39.096227 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl" event={"ID":"c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe","Type":"ContainerDied","Data":"88a9ef8a0775ae8f976f07651cacee1c92ab8a43a1e1ec6d3568edaabe0b534a"} Oct 29 23:25:39.096316 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:39.096253 4821 scope.go:122] "RemoveContainer" containerID="c2422e9076a205fb6c3711163100b56c37c8faca0b2d7d48f4ac613e44633e8c" Oct 29 23:25:39.110389 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:39.110359 4821 scope.go:122] "RemoveContainer" containerID="54363f00966d104348ac4181ee49432d4a9e9f1785ac761321df420926789766" Oct 29 23:25:39.123526 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:39.123504 4821 scope.go:122] "RemoveContainer" containerID="c2422e9076a205fb6c3711163100b56c37c8faca0b2d7d48f4ac613e44633e8c" Oct 29 23:25:39.123996 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:25:39.123975 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2422e9076a205fb6c3711163100b56c37c8faca0b2d7d48f4ac613e44633e8c\": container with ID starting with c2422e9076a205fb6c3711163100b56c37c8faca0b2d7d48f4ac613e44633e8c not found: ID does not exist" containerID="c2422e9076a205fb6c3711163100b56c37c8faca0b2d7d48f4ac613e44633e8c" Oct 29 23:25:39.124083 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:39.124009 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2422e9076a205fb6c3711163100b56c37c8faca0b2d7d48f4ac613e44633e8c"} err="failed to get container status \"c2422e9076a205fb6c3711163100b56c37c8faca0b2d7d48f4ac613e44633e8c\": rpc error: code = NotFound desc = could not find container \"c2422e9076a205fb6c3711163100b56c37c8faca0b2d7d48f4ac613e44633e8c\": container with ID starting with c2422e9076a205fb6c3711163100b56c37c8faca0b2d7d48f4ac613e44633e8c not found: ID does not exist" Oct 29 23:25:39.124083 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:39.124033 4821 scope.go:122] "RemoveContainer" containerID="54363f00966d104348ac4181ee49432d4a9e9f1785ac761321df420926789766" Oct 29 23:25:39.124423 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:25:39.124401 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54363f00966d104348ac4181ee49432d4a9e9f1785ac761321df420926789766\": container with ID starting with 54363f00966d104348ac4181ee49432d4a9e9f1785ac761321df420926789766 not found: ID does not exist" containerID="54363f00966d104348ac4181ee49432d4a9e9f1785ac761321df420926789766" Oct 29 23:25:39.124508 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:39.124430 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54363f00966d104348ac4181ee49432d4a9e9f1785ac761321df420926789766"} err="failed to get container status \"54363f00966d104348ac4181ee49432d4a9e9f1785ac761321df420926789766\": rpc error: code = NotFound desc = could not find container \"54363f00966d104348ac4181ee49432d4a9e9f1785ac761321df420926789766\": container with ID starting with 54363f00966d104348ac4181ee49432d4a9e9f1785ac761321df420926789766 not found: ID does not exist" Oct 29 23:25:39.154113 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:39.154090 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl"] Oct 29 23:25:39.167008 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:39.166985 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f4ce70-openshift-default-665f5b98f9-bwggl"] Oct 29 23:25:39.776957 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:39.776915 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a9ae14-9c947bdc4-tbcvf" podUID="0fd71b7c-e947-4db2-97fb-e16431ec4a37" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.255:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:25:40.008383 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.008013 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk" podUID="e52400e6-b2db-4dca-b23d-ad9f9a81bf44" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:25:40.132672 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.132640 4821 generic.go:356] "Generic (PLEG): container finished" podID="073c5be5-712e-4634-82a1-9f957244d5e1" containerID="be155f2bf566b59089ba78a18a5fb3dc25269005ce0063e5b783a15127bf8e6d" exitCode=0 Oct 29 23:25:40.133181 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.132709 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" event={"ID":"073c5be5-712e-4634-82a1-9f957244d5e1","Type":"ContainerDied","Data":"be155f2bf566b59089ba78a18a5fb3dc25269005ce0063e5b783a15127bf8e6d"} Oct 29 23:25:40.164364 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.164340 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:25:40.171057 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.171029 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" path="/var/lib/kubelet/pods/c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe/volumes" Oct 29 23:25:40.244507 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.244476 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/073c5be5-712e-4634-82a1-9f957244d5e1-istio-token\" (UniqueName: \"kubernetes.io/projected/073c5be5-712e-4634-82a1-9f957244d5e1-istio-token\") pod \"073c5be5-712e-4634-82a1-9f957244d5e1\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " Oct 29 23:25:40.244694 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.244522 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-istio-envoy\") pod \"073c5be5-712e-4634-82a1-9f957244d5e1\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " Oct 29 23:25:40.244694 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.244582 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/073c5be5-712e-4634-82a1-9f957244d5e1-kube-api-access-27gcp\" (UniqueName: \"kubernetes.io/projected/073c5be5-712e-4634-82a1-9f957244d5e1-kube-api-access-27gcp\") pod \"073c5be5-712e-4634-82a1-9f957244d5e1\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " Oct 29 23:25:40.244818 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.244696 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/073c5be5-712e-4634-82a1-9f957244d5e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073c5be5-712e-4634-82a1-9f957244d5e1-istiod-ca-cert\") pod \"073c5be5-712e-4634-82a1-9f957244d5e1\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " Oct 29 23:25:40.244818 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.244789 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-credential-socket\") pod \"073c5be5-712e-4634-82a1-9f957244d5e1\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " Oct 29 23:25:40.244930 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.244843 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/073c5be5-712e-4634-82a1-9f957244d5e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073c5be5-712e-4634-82a1-9f957244d5e1-istio-podinfo\") pod \"073c5be5-712e-4634-82a1-9f957244d5e1\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " Oct 29 23:25:40.244930 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.244878 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-workload-certs\") pod \"073c5be5-712e-4634-82a1-9f957244d5e1\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " Oct 29 23:25:40.245038 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.244979 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-workload-socket\") pod \"073c5be5-712e-4634-82a1-9f957244d5e1\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " Oct 29 23:25:40.245090 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.245062 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-credential-socket" pod "073c5be5-712e-4634-82a1-9f957244d5e1" (UID: "073c5be5-712e-4634-82a1-9f957244d5e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:40.245090 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.245071 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/073c5be5-712e-4634-82a1-9f957244d5e1-istiod-ca-cert" pod "073c5be5-712e-4634-82a1-9f957244d5e1" (UID: "073c5be5-712e-4634-82a1-9f957244d5e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:40.245232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.245096 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-istio-data\") pod \"073c5be5-712e-4634-82a1-9f957244d5e1\" (UID: \"073c5be5-712e-4634-82a1-9f957244d5e1\") " Oct 29 23:25:40.245289 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.245236 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-workload-certs" pod "073c5be5-712e-4634-82a1-9f957244d5e1" (UID: "073c5be5-712e-4634-82a1-9f957244d5e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:40.245406 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.245390 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-workload-socket" pod "073c5be5-712e-4634-82a1-9f957244d5e1" (UID: "073c5be5-712e-4634-82a1-9f957244d5e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:40.245601 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.245579 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-istio-data" pod "073c5be5-712e-4634-82a1-9f957244d5e1" (UID: "073c5be5-712e-4634-82a1-9f957244d5e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:40.246092 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.246024 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:40.246092 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.246093 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073c5be5-712e-4634-82a1-9f957244d5e1-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:40.246311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.246107 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:40.246311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.246123 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:40.246311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.246136 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:40.248327 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.248299 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/073c5be5-712e-4634-82a1-9f957244d5e1-istio-podinfo" pod "073c5be5-712e-4634-82a1-9f957244d5e1" (UID: "073c5be5-712e-4634-82a1-9f957244d5e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:25:40.255956 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.255930 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073c5be5-712e-4634-82a1-9f957244d5e1-istio-token" pod "073c5be5-712e-4634-82a1-9f957244d5e1" (UID: "073c5be5-712e-4634-82a1-9f957244d5e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:40.256118 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.256099 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-istio-envoy" pod "073c5be5-712e-4634-82a1-9f957244d5e1" (UID: "073c5be5-712e-4634-82a1-9f957244d5e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:40.258134 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.258107 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073c5be5-712e-4634-82a1-9f957244d5e1-kube-api-access-27gcp" pod "073c5be5-712e-4634-82a1-9f957244d5e1" (UID: "073c5be5-712e-4634-82a1-9f957244d5e1"). InnerVolumeSpecName "kube-api-access-27gcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:40.349438 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.349398 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073c5be5-712e-4634-82a1-9f957244d5e1-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:40.349438 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.349438 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/073c5be5-712e-4634-82a1-9f957244d5e1-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:40.349682 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.349452 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073c5be5-712e-4634-82a1-9f957244d5e1-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:40.349682 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:40.349466 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27gcp\" (UniqueName: \"kubernetes.io/projected/073c5be5-712e-4634-82a1-9f957244d5e1-kube-api-access-27gcp\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:41.095943 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.095918 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 23:25:41.142687 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.142653 4821 generic.go:356] "Generic (PLEG): container finished" podID="30e8d47c-27fb-40d8-86fb-eea11610630b" containerID="511573a6f4f5a42b2a375ee02c1f383872f7cb22915a0563dd6ef36fef9b9b84" exitCode=0 Oct 29 23:25:41.143120 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.142703 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" Oct 29 23:25:41.143120 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.142734 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" event={"ID":"30e8d47c-27fb-40d8-86fb-eea11610630b","Type":"ContainerDied","Data":"511573a6f4f5a42b2a375ee02c1f383872f7cb22915a0563dd6ef36fef9b9b84"} Oct 29 23:25:41.143120 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.142781 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk" event={"ID":"30e8d47c-27fb-40d8-86fb-eea11610630b","Type":"ContainerDied","Data":"2a8fff725a18be122921d587581475f7fc5d7591b5306d92ae56effd00075110"} Oct 29 23:25:41.143120 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.142802 4821 scope.go:122] "RemoveContainer" containerID="511573a6f4f5a42b2a375ee02c1f383872f7cb22915a0563dd6ef36fef9b9b84" Oct 29 23:25:41.144482 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.144428 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" Oct 29 23:25:41.144482 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.144468 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw" event={"ID":"073c5be5-712e-4634-82a1-9f957244d5e1","Type":"ContainerDied","Data":"8b436f0bef497dd483d313f2816a113eba6a6860f6c52c4bddce8e6c6a50d806"} Oct 29 23:25:41.155013 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.154995 4821 scope.go:122] "RemoveContainer" containerID="511573a6f4f5a42b2a375ee02c1f383872f7cb22915a0563dd6ef36fef9b9b84" Oct 29 23:25:41.155267 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:25:41.155247 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"511573a6f4f5a42b2a375ee02c1f383872f7cb22915a0563dd6ef36fef9b9b84\": container with ID starting with 511573a6f4f5a42b2a375ee02c1f383872f7cb22915a0563dd6ef36fef9b9b84 not found: ID does not exist" containerID="511573a6f4f5a42b2a375ee02c1f383872f7cb22915a0563dd6ef36fef9b9b84" Oct 29 23:25:41.155349 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.155274 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"511573a6f4f5a42b2a375ee02c1f383872f7cb22915a0563dd6ef36fef9b9b84"} err="failed to get container status \"511573a6f4f5a42b2a375ee02c1f383872f7cb22915a0563dd6ef36fef9b9b84\": rpc error: code = NotFound desc = could not find container \"511573a6f4f5a42b2a375ee02c1f383872f7cb22915a0563dd6ef36fef9b9b84\": container with ID starting with 511573a6f4f5a42b2a375ee02c1f383872f7cb22915a0563dd6ef36fef9b9b84 not found: ID does not exist" Oct 29 23:25:41.155349 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.155291 4821 scope.go:122] "RemoveContainer" containerID="be155f2bf566b59089ba78a18a5fb3dc25269005ce0063e5b783a15127bf8e6d" Oct 29 23:25:41.165607 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.165572 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-workload-socket\") pod \"30e8d47c-27fb-40d8-86fb-eea11610630b\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " Oct 29 23:25:41.165784 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.165765 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30e8d47c-27fb-40d8-86fb-eea11610630b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30e8d47c-27fb-40d8-86fb-eea11610630b-istiod-ca-cert\") pod \"30e8d47c-27fb-40d8-86fb-eea11610630b\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " Oct 29 23:25:41.165873 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.165842 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30e8d47c-27fb-40d8-86fb-eea11610630b-kube-api-access-zjxgh\" (UniqueName: \"kubernetes.io/projected/30e8d47c-27fb-40d8-86fb-eea11610630b-kube-api-access-zjxgh\") pod \"30e8d47c-27fb-40d8-86fb-eea11610630b\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " Oct 29 23:25:41.165934 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.165920 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-podinfo\") pod \"30e8d47c-27fb-40d8-86fb-eea11610630b\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " Oct 29 23:25:41.166093 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.166044 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-workload-certs\") pod \"30e8d47c-27fb-40d8-86fb-eea11610630b\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " Oct 29 23:25:41.166168 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.166114 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-envoy\") pod \"30e8d47c-27fb-40d8-86fb-eea11610630b\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " Oct 29 23:25:41.166242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.166149 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-data\") pod \"30e8d47c-27fb-40d8-86fb-eea11610630b\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " Oct 29 23:25:41.166242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.166225 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-token\" (UniqueName: \"kubernetes.io/projected/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-token\") pod \"30e8d47c-27fb-40d8-86fb-eea11610630b\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " Oct 29 23:25:41.166366 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.166288 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-credential-socket\") pod \"30e8d47c-27fb-40d8-86fb-eea11610630b\" (UID: \"30e8d47c-27fb-40d8-86fb-eea11610630b\") " Oct 29 23:25:41.168353 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.168323 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-workload-certs" pod "30e8d47c-27fb-40d8-86fb-eea11610630b" (UID: "30e8d47c-27fb-40d8-86fb-eea11610630b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:41.168673 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.168647 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-data" pod "30e8d47c-27fb-40d8-86fb-eea11610630b" (UID: "30e8d47c-27fb-40d8-86fb-eea11610630b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:41.168990 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.168966 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30e8d47c-27fb-40d8-86fb-eea11610630b-istiod-ca-cert" pod "30e8d47c-27fb-40d8-86fb-eea11610630b" (UID: "30e8d47c-27fb-40d8-86fb-eea11610630b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:41.169234 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.169187 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-workload-socket" pod "30e8d47c-27fb-40d8-86fb-eea11610630b" (UID: "30e8d47c-27fb-40d8-86fb-eea11610630b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:41.170707 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.170681 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30e8d47c-27fb-40d8-86fb-eea11610630b-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:41.170795 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.170711 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:41.170795 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.170725 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:41.170795 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.170740 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:41.170980 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.170958 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-credential-socket" pod "30e8d47c-27fb-40d8-86fb-eea11610630b" (UID: "30e8d47c-27fb-40d8-86fb-eea11610630b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:41.184576 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.184551 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-podinfo" pod "30e8d47c-27fb-40d8-86fb-eea11610630b" (UID: "30e8d47c-27fb-40d8-86fb-eea11610630b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:25:41.186292 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.186263 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-envoy" pod "30e8d47c-27fb-40d8-86fb-eea11610630b" (UID: "30e8d47c-27fb-40d8-86fb-eea11610630b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:41.186395 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.186328 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-token" pod "30e8d47c-27fb-40d8-86fb-eea11610630b" (UID: "30e8d47c-27fb-40d8-86fb-eea11610630b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:41.187581 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.187408 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e8d47c-27fb-40d8-86fb-eea11610630b-kube-api-access-zjxgh" pod "30e8d47c-27fb-40d8-86fb-eea11610630b" (UID: "30e8d47c-27fb-40d8-86fb-eea11610630b"). InnerVolumeSpecName "kube-api-access-zjxgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:41.205085 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.205061 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw"] Oct 29 23:25:41.211679 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.211658 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4504f1-openshift-default-6f87db694c-4c5lw"] Oct 29 23:25:41.272755 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.272716 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:41.272755 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.272753 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:41.272934 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.272763 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e8d47c-27fb-40d8-86fb-eea11610630b-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:41.272934 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.272773 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjxgh\" (UniqueName: \"kubernetes.io/projected/30e8d47c-27fb-40d8-86fb-eea11610630b-kube-api-access-zjxgh\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:41.272934 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.272782 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30e8d47c-27fb-40d8-86fb-eea11610630b-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:41.493263 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.493227 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk"] Oct 29 23:25:41.506322 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:41.506296 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-jvmhk"] Oct 29 23:25:42.171383 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:42.171344 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="073c5be5-712e-4634-82a1-9f957244d5e1" path="/var/lib/kubelet/pods/073c5be5-712e-4634-82a1-9f957244d5e1/volumes" Oct 29 23:25:42.172075 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:42.172053 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30e8d47c-27fb-40d8-86fb-eea11610630b" path="/var/lib/kubelet/pods/30e8d47c-27fb-40d8-86fb-eea11610630b/volumes" Oct 29 23:25:43.066221 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:43.064967 4821 patch_prober.go:29] interesting pod/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:25:43.066221 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:43.065028 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" podUID="da46e2ae-5388-4445-8351-bcc2204a9dcf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:25:43.965114 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:43.965082 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 23:25:44.009680 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.009656 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:25:44.026417 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.026380 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-token\" (UniqueName: \"kubernetes.io/projected/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-token\") pod \"da46e2ae-5388-4445-8351-bcc2204a9dcf\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " Oct 29 23:25:44.026580 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.026453 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-envoy\") pod \"da46e2ae-5388-4445-8351-bcc2204a9dcf\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " Oct 29 23:25:44.026580 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.026552 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da46e2ae-5388-4445-8351-bcc2204a9dcf-kube-api-access-2hx99\" (UniqueName: \"kubernetes.io/projected/da46e2ae-5388-4445-8351-bcc2204a9dcf-kube-api-access-2hx99\") pod \"da46e2ae-5388-4445-8351-bcc2204a9dcf\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " Oct 29 23:25:44.026700 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.026618 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-workload-certs\") pod \"da46e2ae-5388-4445-8351-bcc2204a9dcf\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " Oct 29 23:25:44.026700 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.026659 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da46e2ae-5388-4445-8351-bcc2204a9dcf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da46e2ae-5388-4445-8351-bcc2204a9dcf-istiod-ca-cert\") pod \"da46e2ae-5388-4445-8351-bcc2204a9dcf\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " Oct 29 23:25:44.026808 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.026751 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-podinfo\") pod \"da46e2ae-5388-4445-8351-bcc2204a9dcf\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " Oct 29 23:25:44.026808 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.026781 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-workload-socket\") pod \"da46e2ae-5388-4445-8351-bcc2204a9dcf\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " Oct 29 23:25:44.026907 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.026822 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-data\") pod \"da46e2ae-5388-4445-8351-bcc2204a9dcf\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " Oct 29 23:25:44.026907 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.026879 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-credential-socket\") pod \"da46e2ae-5388-4445-8351-bcc2204a9dcf\" (UID: \"da46e2ae-5388-4445-8351-bcc2204a9dcf\") " Oct 29 23:25:44.027413 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.027361 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-workload-socket" pod "da46e2ae-5388-4445-8351-bcc2204a9dcf" (UID: "da46e2ae-5388-4445-8351-bcc2204a9dcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:44.027601 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.027570 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-credential-socket" pod "da46e2ae-5388-4445-8351-bcc2204a9dcf" (UID: "da46e2ae-5388-4445-8351-bcc2204a9dcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:44.027679 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.027649 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.028019 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.027717 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-data" pod "da46e2ae-5388-4445-8351-bcc2204a9dcf" (UID: "da46e2ae-5388-4445-8351-bcc2204a9dcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:44.028019 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.027735 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-workload-certs" pod "da46e2ae-5388-4445-8351-bcc2204a9dcf" (UID: "da46e2ae-5388-4445-8351-bcc2204a9dcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:44.028019 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.027953 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da46e2ae-5388-4445-8351-bcc2204a9dcf-istiod-ca-cert" pod "da46e2ae-5388-4445-8351-bcc2204a9dcf" (UID: "da46e2ae-5388-4445-8351-bcc2204a9dcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:44.029733 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.029707 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-token" pod "da46e2ae-5388-4445-8351-bcc2204a9dcf" (UID: "da46e2ae-5388-4445-8351-bcc2204a9dcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:44.030857 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.030809 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da46e2ae-5388-4445-8351-bcc2204a9dcf-kube-api-access-2hx99" pod "da46e2ae-5388-4445-8351-bcc2204a9dcf" (UID: "da46e2ae-5388-4445-8351-bcc2204a9dcf"). InnerVolumeSpecName "kube-api-access-2hx99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:44.033751 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.033727 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-podinfo" pod "da46e2ae-5388-4445-8351-bcc2204a9dcf" (UID: "da46e2ae-5388-4445-8351-bcc2204a9dcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:25:44.034995 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.034972 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-envoy" pod "da46e2ae-5388-4445-8351-bcc2204a9dcf" (UID: "da46e2ae-5388-4445-8351-bcc2204a9dcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:44.129094 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.129060 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-data\") pod \"df9f4a39-1043-449e-b5b3-b57c82b618b3\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " Oct 29 23:25:44.129332 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.129138 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-token\" (UniqueName: \"kubernetes.io/projected/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-token\") pod \"df9f4a39-1043-449e-b5b3-b57c82b618b3\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " Oct 29 23:25:44.129332 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.129182 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df9f4a39-1043-449e-b5b3-b57c82b618b3-kube-api-access-vp8z5\" (UniqueName: \"kubernetes.io/projected/df9f4a39-1043-449e-b5b3-b57c82b618b3-kube-api-access-vp8z5\") pod \"df9f4a39-1043-449e-b5b3-b57c82b618b3\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " Oct 29 23:25:44.129332 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.129245 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-podinfo\") pod \"df9f4a39-1043-449e-b5b3-b57c82b618b3\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " Oct 29 23:25:44.129332 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.129283 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-credential-socket\") pod \"df9f4a39-1043-449e-b5b3-b57c82b618b3\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " Oct 29 23:25:44.129619 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.129346 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-workload-socket\") pod \"df9f4a39-1043-449e-b5b3-b57c82b618b3\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " Oct 29 23:25:44.129619 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.129458 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-envoy\") pod \"df9f4a39-1043-449e-b5b3-b57c82b618b3\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " Oct 29 23:25:44.129619 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.129491 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-data" pod "df9f4a39-1043-449e-b5b3-b57c82b618b3" (UID: "df9f4a39-1043-449e-b5b3-b57c82b618b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:44.129619 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.129553 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-workload-certs\") pod \"df9f4a39-1043-449e-b5b3-b57c82b618b3\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " Oct 29 23:25:44.129811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.129652 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df9f4a39-1043-449e-b5b3-b57c82b618b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df9f4a39-1043-449e-b5b3-b57c82b618b3-istiod-ca-cert\") pod \"df9f4a39-1043-449e-b5b3-b57c82b618b3\" (UID: \"df9f4a39-1043-449e-b5b3-b57c82b618b3\") " Oct 29 23:25:44.129811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.129719 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-workload-socket" pod "df9f4a39-1043-449e-b5b3-b57c82b618b3" (UID: "df9f4a39-1043-449e-b5b3-b57c82b618b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:44.129811 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.129749 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-credential-socket" pod "df9f4a39-1043-449e-b5b3-b57c82b618b3" (UID: "df9f4a39-1043-449e-b5b3-b57c82b618b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:44.130734 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.130686 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-workload-certs" pod "df9f4a39-1043-449e-b5b3-b57c82b618b3" (UID: "df9f4a39-1043-449e-b5b3-b57c82b618b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:44.130922 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.130853 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9f4a39-1043-449e-b5b3-b57c82b618b3-istiod-ca-cert" pod "df9f4a39-1043-449e-b5b3-b57c82b618b3" (UID: "df9f4a39-1043-449e-b5b3-b57c82b618b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:44.130922 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.130859 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da46e2ae-5388-4445-8351-bcc2204a9dcf-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.130922 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.130899 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.130922 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.130915 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.131162 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.130931 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.131162 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.130948 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.131162 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.130963 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.131162 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.130975 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.131162 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.130988 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.131162 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.131000 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.131162 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.131013 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hx99\" (UniqueName: \"kubernetes.io/projected/da46e2ae-5388-4445-8351-bcc2204a9dcf-kube-api-access-2hx99\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.131162 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.131026 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da46e2ae-5388-4445-8351-bcc2204a9dcf-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.132072 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.132052 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-podinfo" pod "df9f4a39-1043-449e-b5b3-b57c82b618b3" (UID: "df9f4a39-1043-449e-b5b3-b57c82b618b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:25:44.132189 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.132059 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df9f4a39-1043-449e-b5b3-b57c82b618b3-kube-api-access-vp8z5" pod "df9f4a39-1043-449e-b5b3-b57c82b618b3" (UID: "df9f4a39-1043-449e-b5b3-b57c82b618b3"). InnerVolumeSpecName "kube-api-access-vp8z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:44.132343 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.132321 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-envoy" pod "df9f4a39-1043-449e-b5b3-b57c82b618b3" (UID: "df9f4a39-1043-449e-b5b3-b57c82b618b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:44.133937 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.133912 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-token" pod "df9f4a39-1043-449e-b5b3-b57c82b618b3" (UID: "df9f4a39-1043-449e-b5b3-b57c82b618b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:44.171116 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.171053 4821 generic.go:356] "Generic (PLEG): container finished" podID="df9f4a39-1043-449e-b5b3-b57c82b618b3" containerID="c4948e75f358f5cd5e3cbf70b52b02fa3804855bdb9c464547466161a0df51ef" exitCode=0 Oct 29 23:25:44.171280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.171167 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" Oct 29 23:25:44.172023 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.172000 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" event={"ID":"df9f4a39-1043-449e-b5b3-b57c82b618b3","Type":"ContainerDied","Data":"c4948e75f358f5cd5e3cbf70b52b02fa3804855bdb9c464547466161a0df51ef"} Oct 29 23:25:44.172100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.172044 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp" event={"ID":"df9f4a39-1043-449e-b5b3-b57c82b618b3","Type":"ContainerDied","Data":"23cf309f5201487e83a5893c7932e749b8144cd153be1599fd37b583640d0e51"} Oct 29 23:25:44.172100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.172065 4821 scope.go:122] "RemoveContainer" containerID="c4948e75f358f5cd5e3cbf70b52b02fa3804855bdb9c464547466161a0df51ef" Oct 29 23:25:44.173090 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.173066 4821 generic.go:356] "Generic (PLEG): container finished" podID="da46e2ae-5388-4445-8351-bcc2204a9dcf" containerID="4fc80bbdf569b4c193d1b0fa400f79242cf0a642dc0edaad6ef227e83a9061e0" exitCode=0 Oct 29 23:25:44.173255 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.173117 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" Oct 29 23:25:44.173368 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.173137 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" event={"ID":"da46e2ae-5388-4445-8351-bcc2204a9dcf","Type":"ContainerDied","Data":"4fc80bbdf569b4c193d1b0fa400f79242cf0a642dc0edaad6ef227e83a9061e0"} Oct 29 23:25:44.173499 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.173372 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz" event={"ID":"da46e2ae-5388-4445-8351-bcc2204a9dcf","Type":"ContainerDied","Data":"2679e738eda982de78374d8f8fe5e32c36d7a1903d537c8ebea66e14ba43709b"} Oct 29 23:25:44.192892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.192868 4821 scope.go:122] "RemoveContainer" containerID="c4948e75f358f5cd5e3cbf70b52b02fa3804855bdb9c464547466161a0df51ef" Oct 29 23:25:44.194898 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:25:44.194867 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4948e75f358f5cd5e3cbf70b52b02fa3804855bdb9c464547466161a0df51ef\": container with ID starting with c4948e75f358f5cd5e3cbf70b52b02fa3804855bdb9c464547466161a0df51ef not found: ID does not exist" containerID="c4948e75f358f5cd5e3cbf70b52b02fa3804855bdb9c464547466161a0df51ef" Oct 29 23:25:44.195025 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.194906 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4948e75f358f5cd5e3cbf70b52b02fa3804855bdb9c464547466161a0df51ef"} err="failed to get container status \"c4948e75f358f5cd5e3cbf70b52b02fa3804855bdb9c464547466161a0df51ef\": rpc error: code = NotFound desc = could not find container \"c4948e75f358f5cd5e3cbf70b52b02fa3804855bdb9c464547466161a0df51ef\": container with ID starting with c4948e75f358f5cd5e3cbf70b52b02fa3804855bdb9c464547466161a0df51ef not found: ID does not exist" Oct 29 23:25:44.195025 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.194927 4821 scope.go:122] "RemoveContainer" containerID="4fc80bbdf569b4c193d1b0fa400f79242cf0a642dc0edaad6ef227e83a9061e0" Oct 29 23:25:44.208487 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.208469 4821 scope.go:122] "RemoveContainer" containerID="4fc80bbdf569b4c193d1b0fa400f79242cf0a642dc0edaad6ef227e83a9061e0" Oct 29 23:25:44.208781 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:25:44.208759 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fc80bbdf569b4c193d1b0fa400f79242cf0a642dc0edaad6ef227e83a9061e0\": container with ID starting with 4fc80bbdf569b4c193d1b0fa400f79242cf0a642dc0edaad6ef227e83a9061e0 not found: ID does not exist" containerID="4fc80bbdf569b4c193d1b0fa400f79242cf0a642dc0edaad6ef227e83a9061e0" Oct 29 23:25:44.208842 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.208788 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fc80bbdf569b4c193d1b0fa400f79242cf0a642dc0edaad6ef227e83a9061e0"} err="failed to get container status \"4fc80bbdf569b4c193d1b0fa400f79242cf0a642dc0edaad6ef227e83a9061e0\": rpc error: code = NotFound desc = could not find container \"4fc80bbdf569b4c193d1b0fa400f79242cf0a642dc0edaad6ef227e83a9061e0\": container with ID starting with 4fc80bbdf569b4c193d1b0fa400f79242cf0a642dc0edaad6ef227e83a9061e0 not found: ID does not exist" Oct 29 23:25:44.233114 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.233089 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.233114 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.233115 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df9f4a39-1043-449e-b5b3-b57c82b618b3-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.233267 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.233124 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df9f4a39-1043-449e-b5b3-b57c82b618b3-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.233267 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.233132 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.233267 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.233140 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp8z5\" (UniqueName: \"kubernetes.io/projected/df9f4a39-1043-449e-b5b3-b57c82b618b3-kube-api-access-vp8z5\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.233267 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.233149 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df9f4a39-1043-449e-b5b3-b57c82b618b3-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:44.241406 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.241383 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz"] Oct 29 23:25:44.245273 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.245252 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-gcllz"] Oct 29 23:25:44.280103 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.280079 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp"] Oct 29 23:25:44.287594 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.287566 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-32717a-openshift-default-69bcd68955-wqvbp"] Oct 29 23:25:44.600241 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.600135 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq" podUID="27c4bd01-02f4-4c07-8792-4f880626a8c0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:25:44.709509 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.709470 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f"] Oct 29 23:25:44.712109 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:44.711442 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" podUID="9a986cc4-88d7-432a-8cc8-f2d5371373b1" containerName="istio-proxy" containerID="cri-o://a804ec26806cf5bf0a1417b5334b75b0c170c846ddc48b4d62841ba035b2dccd" gracePeriod=30 Oct 29 23:25:45.213169 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:45.213130 4821 generic.go:356] "Generic (PLEG): container finished" podID="27c4bd01-02f4-4c07-8792-4f880626a8c0" containerID="d741cd0c92a2e9aefdc43d89d9a4b4908b37a9ecff36883fb2eab788a02d9595" exitCode=0 Oct 29 23:25:45.215147 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:45.215110 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq" event={"ID":"27c4bd01-02f4-4c07-8792-4f880626a8c0","Type":"ContainerDied","Data":"d741cd0c92a2e9aefdc43d89d9a4b4908b37a9ecff36883fb2eab788a02d9595"} Oct 29 23:25:45.297279 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:45.297243 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq" Oct 29 23:25:45.374339 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:45.374298 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27c4bd01-02f4-4c07-8792-4f880626a8c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c4bd01-02f4-4c07-8792-4f880626a8c0-plugins-config-volume\") pod \"27c4bd01-02f4-4c07-8792-4f880626a8c0\" (UID: \"27c4bd01-02f4-4c07-8792-4f880626a8c0\") " Oct 29 23:25:45.374514 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:45.374474 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27c4bd01-02f4-4c07-8792-4f880626a8c0-kube-api-access-xncdq\" (UniqueName: \"kubernetes.io/projected/27c4bd01-02f4-4c07-8792-4f880626a8c0-kube-api-access-xncdq\") pod \"27c4bd01-02f4-4c07-8792-4f880626a8c0\" (UID: \"27c4bd01-02f4-4c07-8792-4f880626a8c0\") " Oct 29 23:25:45.374707 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:45.374679 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c4bd01-02f4-4c07-8792-4f880626a8c0-plugins-config-volume" pod "27c4bd01-02f4-4c07-8792-4f880626a8c0" (UID: "27c4bd01-02f4-4c07-8792-4f880626a8c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:45.375013 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:45.374988 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c4bd01-02f4-4c07-8792-4f880626a8c0-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:45.376876 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:45.376850 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c4bd01-02f4-4c07-8792-4f880626a8c0-kube-api-access-xncdq" pod "27c4bd01-02f4-4c07-8792-4f880626a8c0" (UID: "27c4bd01-02f4-4c07-8792-4f880626a8c0"). InnerVolumeSpecName "kube-api-access-xncdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:45.476189 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:45.476157 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xncdq\" (UniqueName: \"kubernetes.io/projected/27c4bd01-02f4-4c07-8792-4f880626a8c0-kube-api-access-xncdq\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:46.171728 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:46.171694 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da46e2ae-5388-4445-8351-bcc2204a9dcf" path="/var/lib/kubelet/pods/da46e2ae-5388-4445-8351-bcc2204a9dcf/volumes" Oct 29 23:25:46.172267 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:46.172250 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df9f4a39-1043-449e-b5b3-b57c82b618b3" path="/var/lib/kubelet/pods/df9f4a39-1043-449e-b5b3-b57c82b618b3/volumes" Oct 29 23:25:46.224137 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:46.224104 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq" event={"ID":"27c4bd01-02f4-4c07-8792-4f880626a8c0","Type":"ContainerDied","Data":"a5291477216430493a0a4556b4016e4b0d6bfaf42439de8c2d981247e2664568"} Oct 29 23:25:46.224137 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:46.224133 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq" Oct 29 23:25:46.224612 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:46.224172 4821 scope.go:122] "RemoveContainer" containerID="d741cd0c92a2e9aefdc43d89d9a4b4908b37a9ecff36883fb2eab788a02d9595" Oct 29 23:25:46.367052 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:46.367018 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq"] Oct 29 23:25:46.380228 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:46.380186 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-faa4fc-56bf8cdc8c-459tq"] Oct 29 23:25:47.663218 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:47.663177 4821 patch_prober.go:29] interesting pod/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:25:47.663596 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:47.663286 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" podUID="9a986cc4-88d7-432a-8cc8-f2d5371373b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:25:47.912408 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:47.912378 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk" Oct 29 23:25:48.013970 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:48.013942 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52400e6-b2db-4dca-b23d-ad9f9a81bf44-kube-api-access-vmkfv\" (UniqueName: \"kubernetes.io/projected/e52400e6-b2db-4dca-b23d-ad9f9a81bf44-kube-api-access-vmkfv\") pod \"e52400e6-b2db-4dca-b23d-ad9f9a81bf44\" (UID: \"e52400e6-b2db-4dca-b23d-ad9f9a81bf44\") " Oct 29 23:25:48.014127 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:48.014045 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e52400e6-b2db-4dca-b23d-ad9f9a81bf44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52400e6-b2db-4dca-b23d-ad9f9a81bf44-plugins-config-volume\") pod \"e52400e6-b2db-4dca-b23d-ad9f9a81bf44\" (UID: \"e52400e6-b2db-4dca-b23d-ad9f9a81bf44\") " Oct 29 23:25:48.014437 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:48.014402 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e52400e6-b2db-4dca-b23d-ad9f9a81bf44-plugins-config-volume" pod "e52400e6-b2db-4dca-b23d-ad9f9a81bf44" (UID: "e52400e6-b2db-4dca-b23d-ad9f9a81bf44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:48.014692 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:48.014675 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52400e6-b2db-4dca-b23d-ad9f9a81bf44-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:48.016226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:48.016188 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52400e6-b2db-4dca-b23d-ad9f9a81bf44-kube-api-access-vmkfv" pod "e52400e6-b2db-4dca-b23d-ad9f9a81bf44" (UID: "e52400e6-b2db-4dca-b23d-ad9f9a81bf44"). InnerVolumeSpecName "kube-api-access-vmkfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:48.116010 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:48.115981 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmkfv\" (UniqueName: \"kubernetes.io/projected/e52400e6-b2db-4dca-b23d-ad9f9a81bf44-kube-api-access-vmkfv\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:48.171964 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:48.171893 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27c4bd01-02f4-4c07-8792-4f880626a8c0" path="/var/lib/kubelet/pods/27c4bd01-02f4-4c07-8792-4f880626a8c0/volumes" Oct 29 23:25:48.244411 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:48.244381 4821 generic.go:356] "Generic (PLEG): container finished" podID="e52400e6-b2db-4dca-b23d-ad9f9a81bf44" containerID="0ed57a09dd98f3049981c1e8c26b799da8836aeb5e34be23f0d3dbaf98895f9d" exitCode=0 Oct 29 23:25:48.244589 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:48.244424 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk" Oct 29 23:25:48.244589 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:48.244458 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk" event={"ID":"e52400e6-b2db-4dca-b23d-ad9f9a81bf44","Type":"ContainerDied","Data":"0ed57a09dd98f3049981c1e8c26b799da8836aeb5e34be23f0d3dbaf98895f9d"} Oct 29 23:25:48.244589 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:48.244519 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk" event={"ID":"e52400e6-b2db-4dca-b23d-ad9f9a81bf44","Type":"ContainerDied","Data":"5843dd4f1ebb97d1c167134b7d8bc9f92b900803a114e003ac36d72e1baa5b61"} Oct 29 23:25:48.244589 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:48.244560 4821 scope.go:122] "RemoveContainer" containerID="0ed57a09dd98f3049981c1e8c26b799da8836aeb5e34be23f0d3dbaf98895f9d" Oct 29 23:25:48.255585 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:48.255564 4821 scope.go:122] "RemoveContainer" containerID="0ed57a09dd98f3049981c1e8c26b799da8836aeb5e34be23f0d3dbaf98895f9d" Oct 29 23:25:48.255866 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:25:48.255848 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ed57a09dd98f3049981c1e8c26b799da8836aeb5e34be23f0d3dbaf98895f9d\": container with ID starting with 0ed57a09dd98f3049981c1e8c26b799da8836aeb5e34be23f0d3dbaf98895f9d not found: ID does not exist" containerID="0ed57a09dd98f3049981c1e8c26b799da8836aeb5e34be23f0d3dbaf98895f9d" Oct 29 23:25:48.255926 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:48.255874 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ed57a09dd98f3049981c1e8c26b799da8836aeb5e34be23f0d3dbaf98895f9d"} err="failed to get container status \"0ed57a09dd98f3049981c1e8c26b799da8836aeb5e34be23f0d3dbaf98895f9d\": rpc error: code = NotFound desc = could not find container \"0ed57a09dd98f3049981c1e8c26b799da8836aeb5e34be23f0d3dbaf98895f9d\": container with ID starting with 0ed57a09dd98f3049981c1e8c26b799da8836aeb5e34be23f0d3dbaf98895f9d not found: ID does not exist" Oct 29 23:25:48.301610 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:48.301583 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk"] Oct 29 23:25:48.305502 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:48.305480 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b2b71e-65c7575979-gqdmk"] Oct 29 23:25:49.635804 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:49.635772 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4"] Oct 29 23:25:49.636354 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:49.635984 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" podUID="ce5e06a7-dc83-4262-a6a4-ead5285a1dd3" containerName="maas-api" containerID="cri-o://f99823ad6c628c1498aa1709251b5efb71b710b9a2de83863ead13fe6f059c84" gracePeriod=30 Oct 29 23:25:49.803151 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:49.803109 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" podUID="ce5e06a7-dc83-4262-a6a4-ead5285a1dd3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.243:8443/health\": dial tcp 10.133.0.243:8443: connect: connection refused" Oct 29 23:25:50.297665 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.290911 4821 generic.go:356] "Generic (PLEG): container finished" podID="ce5e06a7-dc83-4262-a6a4-ead5285a1dd3" containerID="f99823ad6c628c1498aa1709251b5efb71b710b9a2de83863ead13fe6f059c84" exitCode=0 Oct 29 23:25:50.297665 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.294071 4821 generic.go:356] "Generic (PLEG): container finished" podID="9a986cc4-88d7-432a-8cc8-f2d5371373b1" containerID="a804ec26806cf5bf0a1417b5334b75b0c170c846ddc48b4d62841ba035b2dccd" exitCode=0 Oct 29 23:25:50.362502 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.362466 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e52400e6-b2db-4dca-b23d-ad9f9a81bf44" path="/var/lib/kubelet/pods/e52400e6-b2db-4dca-b23d-ad9f9a81bf44/volumes" Oct 29 23:25:50.363219 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.363173 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" event={"ID":"ce5e06a7-dc83-4262-a6a4-ead5285a1dd3","Type":"ContainerDied","Data":"f99823ad6c628c1498aa1709251b5efb71b710b9a2de83863ead13fe6f059c84"} Oct 29 23:25:50.363726 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.363703 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" event={"ID":"9a986cc4-88d7-432a-8cc8-f2d5371373b1","Type":"ContainerDied","Data":"a804ec26806cf5bf0a1417b5334b75b0c170c846ddc48b4d62841ba035b2dccd"} Oct 29 23:25:50.368385 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.368367 4821 scope.go:122] "RemoveContainer" containerID="c1cdf8455f07181d0ffbed0d991db9b77bbcef63bb87b23087abf6a61b0404ef" Oct 29 23:25:50.413061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.413041 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" Oct 29 23:25:50.434844 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.434813 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-kube-api-access-sfr4w\" (UniqueName: \"kubernetes.io/projected/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-kube-api-access-sfr4w\") pod \"ce5e06a7-dc83-4262-a6a4-ead5285a1dd3\" (UID: \"ce5e06a7-dc83-4262-a6a4-ead5285a1dd3\") " Oct 29 23:25:50.434844 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.434845 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-openshift-service-ca\") pod \"ce5e06a7-dc83-4262-a6a4-ead5285a1dd3\" (UID: \"ce5e06a7-dc83-4262-a6a4-ead5285a1dd3\") " Oct 29 23:25:50.435057 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.435044 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-maas-api-tls\") pod \"ce5e06a7-dc83-4262-a6a4-ead5285a1dd3\" (UID: \"ce5e06a7-dc83-4262-a6a4-ead5285a1dd3\") " Oct 29 23:25:50.435141 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.435079 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-metrics-certs\") pod \"ce5e06a7-dc83-4262-a6a4-ead5285a1dd3\" (UID: \"ce5e06a7-dc83-4262-a6a4-ead5285a1dd3\") " Oct 29 23:25:50.435253 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.435224 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-openshift-service-ca" pod "ce5e06a7-dc83-4262-a6a4-ead5285a1dd3" (UID: "ce5e06a7-dc83-4262-a6a4-ead5285a1dd3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:50.436052 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.436029 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:50.438488 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.438464 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-maas-api-tls" pod "ce5e06a7-dc83-4262-a6a4-ead5285a1dd3" (UID: "ce5e06a7-dc83-4262-a6a4-ead5285a1dd3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:25:50.438488 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.438478 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-kube-api-access-sfr4w" pod "ce5e06a7-dc83-4262-a6a4-ead5285a1dd3" (UID: "ce5e06a7-dc83-4262-a6a4-ead5285a1dd3"). InnerVolumeSpecName "kube-api-access-sfr4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:50.438650 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.438555 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-metrics-certs" pod "ce5e06a7-dc83-4262-a6a4-ead5285a1dd3" (UID: "ce5e06a7-dc83-4262-a6a4-ead5285a1dd3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:25:50.473865 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.473835 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:25:50.537090 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.537060 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-workload-socket\") pod \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " Oct 29 23:25:50.537090 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.537095 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-data\") pod \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " Oct 29 23:25:50.537345 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.537163 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-workload-certs\") pod \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " Oct 29 23:25:50.537345 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.537229 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-credential-socket\") pod \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " Oct 29 23:25:50.537345 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.537314 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-envoy\") pod \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " Oct 29 23:25:50.537530 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.537387 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-podinfo\") pod \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " Oct 29 23:25:50.537530 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.537448 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-data" pod "9a986cc4-88d7-432a-8cc8-f2d5371373b1" (UID: "9a986cc4-88d7-432a-8cc8-f2d5371373b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:50.537530 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.537454 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istiod-ca-cert\") pod \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " Oct 29 23:25:50.537701 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.537461 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-workload-socket" pod "9a986cc4-88d7-432a-8cc8-f2d5371373b1" (UID: "9a986cc4-88d7-432a-8cc8-f2d5371373b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:50.537701 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.537611 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-credential-socket" pod "9a986cc4-88d7-432a-8cc8-f2d5371373b1" (UID: "9a986cc4-88d7-432a-8cc8-f2d5371373b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:50.537701 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.537652 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a986cc4-88d7-432a-8cc8-f2d5371373b1-kube-api-access-7fr6f\" (UniqueName: \"kubernetes.io/projected/9a986cc4-88d7-432a-8cc8-f2d5371373b1-kube-api-access-7fr6f\") pod \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " Oct 29 23:25:50.537862 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.537703 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-token\" (UniqueName: \"kubernetes.io/projected/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-token\") pod \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\" (UID: \"9a986cc4-88d7-432a-8cc8-f2d5371373b1\") " Oct 29 23:25:50.537862 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.537748 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-workload-certs" pod "9a986cc4-88d7-432a-8cc8-f2d5371373b1" (UID: "9a986cc4-88d7-432a-8cc8-f2d5371373b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:50.537862 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.537755 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istiod-ca-cert" pod "9a986cc4-88d7-432a-8cc8-f2d5371373b1" (UID: "9a986cc4-88d7-432a-8cc8-f2d5371373b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:25:50.538614 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.538336 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:50.538614 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.538503 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:50.538614 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.538517 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:50.538614 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.538530 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfr4w\" (UniqueName: \"kubernetes.io/projected/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3-kube-api-access-sfr4w\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:50.538614 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.538544 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:50.538614 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.538559 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:50.538614 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.538573 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:50.538614 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.538584 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:50.540121 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.540096 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-envoy" pod "9a986cc4-88d7-432a-8cc8-f2d5371373b1" (UID: "9a986cc4-88d7-432a-8cc8-f2d5371373b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:25:50.540296 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.540279 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-podinfo" pod "9a986cc4-88d7-432a-8cc8-f2d5371373b1" (UID: "9a986cc4-88d7-432a-8cc8-f2d5371373b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:25:50.540462 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.540446 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-token" pod "9a986cc4-88d7-432a-8cc8-f2d5371373b1" (UID: "9a986cc4-88d7-432a-8cc8-f2d5371373b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:50.540508 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.540470 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a986cc4-88d7-432a-8cc8-f2d5371373b1-kube-api-access-7fr6f" pod "9a986cc4-88d7-432a-8cc8-f2d5371373b1" (UID: "9a986cc4-88d7-432a-8cc8-f2d5371373b1"). InnerVolumeSpecName "kube-api-access-7fr6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:25:50.640146 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.640111 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fr6f\" (UniqueName: \"kubernetes.io/projected/9a986cc4-88d7-432a-8cc8-f2d5371373b1-kube-api-access-7fr6f\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:50.640146 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.640143 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:50.640146 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.640153 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:50.640632 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:50.640162 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a986cc4-88d7-432a-8cc8-f2d5371373b1-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:25:51.302899 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:51.302876 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" Oct 29 23:25:51.303102 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:51.302873 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4" event={"ID":"ce5e06a7-dc83-4262-a6a4-ead5285a1dd3","Type":"ContainerDied","Data":"8b0a38d30f6631516a0b9381b0e5c29fa4ed3319c4b6067cb16d6dab9d5403aa"} Oct 29 23:25:51.303102 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:51.303005 4821 scope.go:122] "RemoveContainer" containerID="f99823ad6c628c1498aa1709251b5efb71b710b9a2de83863ead13fe6f059c84" Oct 29 23:25:51.304503 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:51.304480 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" Oct 29 23:25:51.304617 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:51.304525 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f" event={"ID":"9a986cc4-88d7-432a-8cc8-f2d5371373b1","Type":"ContainerDied","Data":"19454a3fc92fd8912b38a0c83675e18bdb5db6e0d03940e0191a546200cbecdf"} Oct 29 23:25:51.322186 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:51.322145 4821 scope.go:122] "RemoveContainer" containerID="a804ec26806cf5bf0a1417b5334b75b0c170c846ddc48b4d62841ba035b2dccd" Oct 29 23:25:51.360718 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:51.360691 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f"] Oct 29 23:25:51.373847 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:51.373816 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a9ae14-openshift-default-5c4d8d9fd7-8lg5f"] Oct 29 23:25:51.410113 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:51.410084 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4"] Oct 29 23:25:51.415029 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:51.415004 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-faa4fc-5468d6865f-vbbw4"] Oct 29 23:25:52.175666 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:52.175630 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a986cc4-88d7-432a-8cc8-f2d5371373b1" path="/var/lib/kubelet/pods/9a986cc4-88d7-432a-8cc8-f2d5371373b1/volumes" Oct 29 23:25:52.176308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:25:52.176287 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce5e06a7-dc83-4262-a6a4-ead5285a1dd3" path="/var/lib/kubelet/pods/ce5e06a7-dc83-4262-a6a4-ead5285a1dd3/volumes" Oct 29 23:25:53.888026 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:25:53.887984 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:26:00.778767 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:00.778732 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn"] Oct 29 23:26:00.779474 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:00.778996 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn" podUID="8314c4e0-da83-4173-bc1a-7fb974a7ab36" containerName="maas-api" containerID="cri-o://5b3b7ae6ca738c3a5450687bad73204b7edb87cdc6a159a3a1c1d55917ea0c6c" gracePeriod=30 Oct 29 23:26:01.345080 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:01.344737 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn" Oct 29 23:26:01.390225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:01.386320 4821 generic.go:356] "Generic (PLEG): container finished" podID="8314c4e0-da83-4173-bc1a-7fb974a7ab36" containerID="5b3b7ae6ca738c3a5450687bad73204b7edb87cdc6a159a3a1c1d55917ea0c6c" exitCode=0 Oct 29 23:26:01.390225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:01.386535 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn" event={"ID":"8314c4e0-da83-4173-bc1a-7fb974a7ab36","Type":"ContainerDied","Data":"5b3b7ae6ca738c3a5450687bad73204b7edb87cdc6a159a3a1c1d55917ea0c6c"} Oct 29 23:26:01.390225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:01.386578 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn" event={"ID":"8314c4e0-da83-4173-bc1a-7fb974a7ab36","Type":"ContainerDied","Data":"ca00e4942c41677de40d78468fda3a70ff375dfb94d9091bfb5e23f936977859"} Oct 29 23:26:01.390225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:01.386603 4821 scope.go:122] "RemoveContainer" containerID="5b3b7ae6ca738c3a5450687bad73204b7edb87cdc6a159a3a1c1d55917ea0c6c" Oct 29 23:26:01.390225 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:01.386718 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn" Oct 29 23:26:01.416731 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:01.416704 4821 scope.go:122] "RemoveContainer" containerID="5b3b7ae6ca738c3a5450687bad73204b7edb87cdc6a159a3a1c1d55917ea0c6c" Oct 29 23:26:01.417784 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:26:01.417755 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b3b7ae6ca738c3a5450687bad73204b7edb87cdc6a159a3a1c1d55917ea0c6c\": container with ID starting with 5b3b7ae6ca738c3a5450687bad73204b7edb87cdc6a159a3a1c1d55917ea0c6c not found: ID does not exist" containerID="5b3b7ae6ca738c3a5450687bad73204b7edb87cdc6a159a3a1c1d55917ea0c6c" Oct 29 23:26:01.420458 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:01.417939 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3b7ae6ca738c3a5450687bad73204b7edb87cdc6a159a3a1c1d55917ea0c6c"} err="failed to get container status \"5b3b7ae6ca738c3a5450687bad73204b7edb87cdc6a159a3a1c1d55917ea0c6c\": rpc error: code = NotFound desc = could not find container \"5b3b7ae6ca738c3a5450687bad73204b7edb87cdc6a159a3a1c1d55917ea0c6c\": container with ID starting with 5b3b7ae6ca738c3a5450687bad73204b7edb87cdc6a159a3a1c1d55917ea0c6c not found: ID does not exist" Oct 29 23:26:01.501675 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:01.501584 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8314c4e0-da83-4173-bc1a-7fb974a7ab36-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8314c4e0-da83-4173-bc1a-7fb974a7ab36-metrics-certs\") pod \"8314c4e0-da83-4173-bc1a-7fb974a7ab36\" (UID: \"8314c4e0-da83-4173-bc1a-7fb974a7ab36\") " Oct 29 23:26:01.501857 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:01.501688 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8314c4e0-da83-4173-bc1a-7fb974a7ab36-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8314c4e0-da83-4173-bc1a-7fb974a7ab36-maas-api-tls\") pod \"8314c4e0-da83-4173-bc1a-7fb974a7ab36\" (UID: \"8314c4e0-da83-4173-bc1a-7fb974a7ab36\") " Oct 29 23:26:01.501857 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:01.501776 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8314c4e0-da83-4173-bc1a-7fb974a7ab36-kube-api-access-zwbv5\" (UniqueName: \"kubernetes.io/projected/8314c4e0-da83-4173-bc1a-7fb974a7ab36-kube-api-access-zwbv5\") pod \"8314c4e0-da83-4173-bc1a-7fb974a7ab36\" (UID: \"8314c4e0-da83-4173-bc1a-7fb974a7ab36\") " Oct 29 23:26:01.501857 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:01.501845 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8314c4e0-da83-4173-bc1a-7fb974a7ab36-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8314c4e0-da83-4173-bc1a-7fb974a7ab36-openshift-service-ca\") pod \"8314c4e0-da83-4173-bc1a-7fb974a7ab36\" (UID: \"8314c4e0-da83-4173-bc1a-7fb974a7ab36\") " Oct 29 23:26:01.502464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:01.502436 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8314c4e0-da83-4173-bc1a-7fb974a7ab36-openshift-service-ca" pod "8314c4e0-da83-4173-bc1a-7fb974a7ab36" (UID: "8314c4e0-da83-4173-bc1a-7fb974a7ab36"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:01.502634 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:01.502617 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8314c4e0-da83-4173-bc1a-7fb974a7ab36-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:01.504928 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:01.504885 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8314c4e0-da83-4173-bc1a-7fb974a7ab36-maas-api-tls" pod "8314c4e0-da83-4173-bc1a-7fb974a7ab36" (UID: "8314c4e0-da83-4173-bc1a-7fb974a7ab36"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:26:01.507165 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:01.507131 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8314c4e0-da83-4173-bc1a-7fb974a7ab36-kube-api-access-zwbv5" pod "8314c4e0-da83-4173-bc1a-7fb974a7ab36" (UID: "8314c4e0-da83-4173-bc1a-7fb974a7ab36"). InnerVolumeSpecName "kube-api-access-zwbv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:01.516784 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:01.516747 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8314c4e0-da83-4173-bc1a-7fb974a7ab36-metrics-certs" pod "8314c4e0-da83-4173-bc1a-7fb974a7ab36" (UID: "8314c4e0-da83-4173-bc1a-7fb974a7ab36"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:26:01.604019 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:01.603982 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwbv5\" (UniqueName: \"kubernetes.io/projected/8314c4e0-da83-4173-bc1a-7fb974a7ab36-kube-api-access-zwbv5\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:01.604019 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:01.604012 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8314c4e0-da83-4173-bc1a-7fb974a7ab36-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:01.604019 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:01.604021 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8314c4e0-da83-4173-bc1a-7fb974a7ab36-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:01.741977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:01.741941 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn"] Oct 29 23:26:01.764452 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:01.764320 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b2b71e-7756895c6b-6t2kn"] Oct 29 23:26:02.171186 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:02.171153 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8314c4e0-da83-4173-bc1a-7fb974a7ab36" path="/var/lib/kubelet/pods/8314c4e0-da83-4173-bc1a-7fb974a7ab36/volumes" Oct 29 23:26:08.829223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:08.829158 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5"] Oct 29 23:26:08.829651 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:08.829402 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" podUID="cf2ae83e-0943-4642-aa5f-2378b18bfe9d" containerName="istio-proxy" containerID="cri-o://4142e8fc2469283bcbc13d35d144251eafec43077d73b35b9a5c0120f9dadcd5" gracePeriod=30 Oct 29 23:26:14.311719 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.311687 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:26:14.465577 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.465479 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-kube-api-access-ct7nx\" (UniqueName: \"kubernetes.io/projected/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-kube-api-access-ct7nx\") pod \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " Oct 29 23:26:14.465577 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.465541 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-token\" (UniqueName: \"kubernetes.io/projected/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-token\") pod \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " Oct 29 23:26:14.465834 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.465596 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-workload-socket\") pod \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " Oct 29 23:26:14.465834 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.465670 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-credential-socket\") pod \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " Oct 29 23:26:14.465834 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.465741 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-podinfo\") pod \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " Oct 29 23:26:14.466019 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.465902 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istiod-ca-cert\") pod \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " Oct 29 23:26:14.466019 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.465949 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-envoy\") pod \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " Oct 29 23:26:14.466019 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.465987 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-workload-certs\") pod \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " Oct 29 23:26:14.466194 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.466032 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-data\") pod \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\" (UID: \"cf2ae83e-0943-4642-aa5f-2378b18bfe9d\") " Oct 29 23:26:14.466377 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.466260 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-credential-socket" pod "cf2ae83e-0943-4642-aa5f-2378b18bfe9d" (UID: "cf2ae83e-0943-4642-aa5f-2378b18bfe9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:26:14.466377 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.466322 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istiod-ca-cert" pod "cf2ae83e-0943-4642-aa5f-2378b18bfe9d" (UID: "cf2ae83e-0943-4642-aa5f-2378b18bfe9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:14.466583 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.466558 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-workload-certs" pod "cf2ae83e-0943-4642-aa5f-2378b18bfe9d" (UID: "cf2ae83e-0943-4642-aa5f-2378b18bfe9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:26:14.466777 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.466748 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-data" pod "cf2ae83e-0943-4642-aa5f-2378b18bfe9d" (UID: "cf2ae83e-0943-4642-aa5f-2378b18bfe9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:26:14.466908 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.466634 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-workload-socket" pod "cf2ae83e-0943-4642-aa5f-2378b18bfe9d" (UID: "cf2ae83e-0943-4642-aa5f-2378b18bfe9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:26:14.466908 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.466889 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:14.467043 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.466911 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:14.467043 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.466926 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:14.468792 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.468763 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-token" pod "cf2ae83e-0943-4642-aa5f-2378b18bfe9d" (UID: "cf2ae83e-0943-4642-aa5f-2378b18bfe9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:14.469340 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.469315 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-podinfo" pod "cf2ae83e-0943-4642-aa5f-2378b18bfe9d" (UID: "cf2ae83e-0943-4642-aa5f-2378b18bfe9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:26:14.469501 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.469477 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-envoy" pod "cf2ae83e-0943-4642-aa5f-2378b18bfe9d" (UID: "cf2ae83e-0943-4642-aa5f-2378b18bfe9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:26:14.471122 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.471098 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-kube-api-access-ct7nx" pod "cf2ae83e-0943-4642-aa5f-2378b18bfe9d" (UID: "cf2ae83e-0943-4642-aa5f-2378b18bfe9d"). InnerVolumeSpecName "kube-api-access-ct7nx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:14.518382 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.518342 4821 generic.go:356] "Generic (PLEG): container finished" podID="cf2ae83e-0943-4642-aa5f-2378b18bfe9d" containerID="4142e8fc2469283bcbc13d35d144251eafec43077d73b35b9a5c0120f9dadcd5" exitCode=0 Oct 29 23:26:14.518569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.518408 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" Oct 29 23:26:14.518569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.518424 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" event={"ID":"cf2ae83e-0943-4642-aa5f-2378b18bfe9d","Type":"ContainerDied","Data":"4142e8fc2469283bcbc13d35d144251eafec43077d73b35b9a5c0120f9dadcd5"} Oct 29 23:26:14.518569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.518488 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5" event={"ID":"cf2ae83e-0943-4642-aa5f-2378b18bfe9d","Type":"ContainerDied","Data":"d6eb9b4554bdace69af8ea372fce54b55f494af01c72c6f92c20e973bddea0f9"} Oct 29 23:26:14.518569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.518520 4821 scope.go:122] "RemoveContainer" containerID="4142e8fc2469283bcbc13d35d144251eafec43077d73b35b9a5c0120f9dadcd5" Oct 29 23:26:14.533385 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.533364 4821 scope.go:122] "RemoveContainer" containerID="f14eeccb9ce53195c36db0341645d92bdfbb84d24077d5e4f8723b67cc84a96d" Oct 29 23:26:14.548039 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.548009 4821 scope.go:122] "RemoveContainer" containerID="4142e8fc2469283bcbc13d35d144251eafec43077d73b35b9a5c0120f9dadcd5" Oct 29 23:26:14.548448 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:26:14.548427 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4142e8fc2469283bcbc13d35d144251eafec43077d73b35b9a5c0120f9dadcd5\": container with ID starting with 4142e8fc2469283bcbc13d35d144251eafec43077d73b35b9a5c0120f9dadcd5 not found: ID does not exist" containerID="4142e8fc2469283bcbc13d35d144251eafec43077d73b35b9a5c0120f9dadcd5" Oct 29 23:26:14.548610 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.548456 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4142e8fc2469283bcbc13d35d144251eafec43077d73b35b9a5c0120f9dadcd5"} err="failed to get container status \"4142e8fc2469283bcbc13d35d144251eafec43077d73b35b9a5c0120f9dadcd5\": rpc error: code = NotFound desc = could not find container \"4142e8fc2469283bcbc13d35d144251eafec43077d73b35b9a5c0120f9dadcd5\": container with ID starting with 4142e8fc2469283bcbc13d35d144251eafec43077d73b35b9a5c0120f9dadcd5 not found: ID does not exist" Oct 29 23:26:14.548610 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.548472 4821 scope.go:122] "RemoveContainer" containerID="f14eeccb9ce53195c36db0341645d92bdfbb84d24077d5e4f8723b67cc84a96d" Oct 29 23:26:14.548771 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:26:14.548752 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f14eeccb9ce53195c36db0341645d92bdfbb84d24077d5e4f8723b67cc84a96d\": container with ID starting with f14eeccb9ce53195c36db0341645d92bdfbb84d24077d5e4f8723b67cc84a96d not found: ID does not exist" containerID="f14eeccb9ce53195c36db0341645d92bdfbb84d24077d5e4f8723b67cc84a96d" Oct 29 23:26:14.548835 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.548775 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f14eeccb9ce53195c36db0341645d92bdfbb84d24077d5e4f8723b67cc84a96d"} err="failed to get container status \"f14eeccb9ce53195c36db0341645d92bdfbb84d24077d5e4f8723b67cc84a96d\": rpc error: code = NotFound desc = could not find container \"f14eeccb9ce53195c36db0341645d92bdfbb84d24077d5e4f8723b67cc84a96d\": container with ID starting with f14eeccb9ce53195c36db0341645d92bdfbb84d24077d5e4f8723b67cc84a96d not found: ID does not exist" Oct 29 23:26:14.568397 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.568372 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct7nx\" (UniqueName: \"kubernetes.io/projected/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-kube-api-access-ct7nx\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:14.568397 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.568398 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:14.568623 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.568408 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:14.568623 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.568417 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:14.568623 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.568425 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:14.568623 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.568435 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2ae83e-0943-4642-aa5f-2378b18bfe9d-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:14.582175 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.582147 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5"] Oct 29 23:26:14.592191 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:14.592162 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b2b71e-openshift-default-56d98c54cf-8qmj5"] Oct 29 23:26:16.702314 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:16.702246 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf2ae83e-0943-4642-aa5f-2378b18bfe9d" path="/var/lib/kubelet/pods/cf2ae83e-0943-4642-aa5f-2378b18bfe9d/volumes" Oct 29 23:26:18.516510 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:18.516475 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm"] Oct 29 23:26:23.890084 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:26:23.890044 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:26:32.308245 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:32.300965 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4"] Oct 29 23:26:32.308245 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:32.301228 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" podUID="49ae9910-d400-4cec-b171-d15541fa2789" containerName="maas-api" containerID="cri-o://8cbc885b699a94e28f961b7c29687941b18824cf7cf5210919f4f11ce24ba465" gracePeriod=30 Oct 29 23:26:32.842399 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:32.841324 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" podUID="49ae9910-d400-4cec-b171-d15541fa2789" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.1.0:8443/health\": dial tcp 10.133.1.0:8443: connect: connection refused" Oct 29 23:26:32.930232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:32.930180 4821 generic.go:356] "Generic (PLEG): container finished" podID="49ae9910-d400-4cec-b171-d15541fa2789" containerID="8cbc885b699a94e28f961b7c29687941b18824cf7cf5210919f4f11ce24ba465" exitCode=0 Oct 29 23:26:32.931429 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:32.930291 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" event={"ID":"49ae9910-d400-4cec-b171-d15541fa2789","Type":"ContainerDied","Data":"8cbc885b699a94e28f961b7c29687941b18824cf7cf5210919f4f11ce24ba465"} Oct 29 23:26:33.014110 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.013886 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" Oct 29 23:26:33.157074 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.156966 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49ae9910-d400-4cec-b171-d15541fa2789-metrics-certs\" (UniqueName: \"kubernetes.io/secret/49ae9910-d400-4cec-b171-d15541fa2789-metrics-certs\") pod \"49ae9910-d400-4cec-b171-d15541fa2789\" (UID: \"49ae9910-d400-4cec-b171-d15541fa2789\") " Oct 29 23:26:33.157074 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.157000 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49ae9910-d400-4cec-b171-d15541fa2789-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49ae9910-d400-4cec-b171-d15541fa2789-maas-api-tls\") pod \"49ae9910-d400-4cec-b171-d15541fa2789\" (UID: \"49ae9910-d400-4cec-b171-d15541fa2789\") " Oct 29 23:26:33.157074 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.157066 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ae9910-d400-4cec-b171-d15541fa2789-kube-api-access-2p5jp\" (UniqueName: \"kubernetes.io/projected/49ae9910-d400-4cec-b171-d15541fa2789-kube-api-access-2p5jp\") pod \"49ae9910-d400-4cec-b171-d15541fa2789\" (UID: \"49ae9910-d400-4cec-b171-d15541fa2789\") " Oct 29 23:26:33.157964 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.157174 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49ae9910-d400-4cec-b171-d15541fa2789-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ae9910-d400-4cec-b171-d15541fa2789-openshift-service-ca\") pod \"49ae9910-d400-4cec-b171-d15541fa2789\" (UID: \"49ae9910-d400-4cec-b171-d15541fa2789\") " Oct 29 23:26:33.157964 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.157629 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ae9910-d400-4cec-b171-d15541fa2789-openshift-service-ca" pod "49ae9910-d400-4cec-b171-d15541fa2789" (UID: "49ae9910-d400-4cec-b171-d15541fa2789"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:33.158255 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.158231 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ae9910-d400-4cec-b171-d15541fa2789-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:33.159907 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.159882 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ae9910-d400-4cec-b171-d15541fa2789-kube-api-access-2p5jp" pod "49ae9910-d400-4cec-b171-d15541fa2789" (UID: "49ae9910-d400-4cec-b171-d15541fa2789"). InnerVolumeSpecName "kube-api-access-2p5jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:33.160032 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.160015 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49ae9910-d400-4cec-b171-d15541fa2789-metrics-certs" pod "49ae9910-d400-4cec-b171-d15541fa2789" (UID: "49ae9910-d400-4cec-b171-d15541fa2789"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:26:33.160099 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.160040 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49ae9910-d400-4cec-b171-d15541fa2789-maas-api-tls" pod "49ae9910-d400-4cec-b171-d15541fa2789" (UID: "49ae9910-d400-4cec-b171-d15541fa2789"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:26:33.260605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.260575 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49ae9910-d400-4cec-b171-d15541fa2789-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:33.260605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.260607 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49ae9910-d400-4cec-b171-d15541fa2789-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:33.260798 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.260623 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p5jp\" (UniqueName: \"kubernetes.io/projected/49ae9910-d400-4cec-b171-d15541fa2789-kube-api-access-2p5jp\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:33.539791 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.539690 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" podUID="cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a" containerName="main" containerID="cri-o://44e235dcf62f1877e02cc0cce5db59c0a3e0107fe6f592e0d115df5b63704cf3" gracePeriod=45 Oct 29 23:26:33.812320 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.812294 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:26:33.940400 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.940363 4821 generic.go:356] "Generic (PLEG): container finished" podID="cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a" containerID="44e235dcf62f1877e02cc0cce5db59c0a3e0107fe6f592e0d115df5b63704cf3" exitCode=0 Oct 29 23:26:33.940587 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.940408 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" Oct 29 23:26:33.940587 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.940442 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" event={"ID":"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a","Type":"ContainerDied","Data":"44e235dcf62f1877e02cc0cce5db59c0a3e0107fe6f592e0d115df5b63704cf3"} Oct 29 23:26:33.940587 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.940491 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm" event={"ID":"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a","Type":"ContainerDied","Data":"7005c4b5f1c0bafde691df63e44cc302fe55aa8fe52c755150a4e7f1497931db"} Oct 29 23:26:33.940587 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.940513 4821 scope.go:122] "RemoveContainer" containerID="44e235dcf62f1877e02cc0cce5db59c0a3e0107fe6f592e0d115df5b63704cf3" Oct 29 23:26:33.942098 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.942080 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" Oct 29 23:26:33.942267 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.942116 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4" event={"ID":"49ae9910-d400-4cec-b171-d15541fa2789","Type":"ContainerDied","Data":"a678b9b58f0ea6ce66d1fb9be0836d9705886e0b9eff785dfcced87031151c4d"} Oct 29 23:26:33.952279 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.952250 4821 scope.go:122] "RemoveContainer" containerID="44e235dcf62f1877e02cc0cce5db59c0a3e0107fe6f592e0d115df5b63704cf3" Oct 29 23:26:33.952565 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:26:33.952547 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44e235dcf62f1877e02cc0cce5db59c0a3e0107fe6f592e0d115df5b63704cf3\": container with ID starting with 44e235dcf62f1877e02cc0cce5db59c0a3e0107fe6f592e0d115df5b63704cf3 not found: ID does not exist" containerID="44e235dcf62f1877e02cc0cce5db59c0a3e0107fe6f592e0d115df5b63704cf3" Oct 29 23:26:33.952635 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.952574 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44e235dcf62f1877e02cc0cce5db59c0a3e0107fe6f592e0d115df5b63704cf3"} err="failed to get container status \"44e235dcf62f1877e02cc0cce5db59c0a3e0107fe6f592e0d115df5b63704cf3\": rpc error: code = NotFound desc = could not find container \"44e235dcf62f1877e02cc0cce5db59c0a3e0107fe6f592e0d115df5b63704cf3\": container with ID starting with 44e235dcf62f1877e02cc0cce5db59c0a3e0107fe6f592e0d115df5b63704cf3 not found: ID does not exist" Oct 29 23:26:33.952635 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.952590 4821 scope.go:122] "RemoveContainer" containerID="8cbc885b699a94e28f961b7c29687941b18824cf7cf5210919f4f11ce24ba465" Oct 29 23:26:33.970398 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.970380 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-kube-api-access-7vlqz\" (UniqueName: \"kubernetes.io/projected/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-kube-api-access-7vlqz\") pod \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\" (UID: \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\") " Oct 29 23:26:33.970530 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.970497 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-model-cache\") pod \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\" (UID: \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\") " Oct 29 23:26:33.970673 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.970657 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-tmp-dir\") pod \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\" (UID: \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\") " Oct 29 23:26:33.970735 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.970721 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-dshm\") pod \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\" (UID: \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\") " Oct 29 23:26:33.970825 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.970811 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-tls-certs\" (UniqueName: \"kubernetes.io/secret/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-tls-certs\") pod \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\" (UID: \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\") " Oct 29 23:26:33.970874 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.970723 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-model-cache" pod "cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a" (UID: "cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:26:33.970874 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.970825 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-tmp-dir" pod "cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a" (UID: "cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:26:33.970953 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.970880 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-home\" (UniqueName: \"kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-home\") pod \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\" (UID: \"cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a\") " Oct 29 23:26:33.971232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.971181 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-home" pod "cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a" (UID: "cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:26:33.971425 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.971404 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:33.971425 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.971422 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:33.971620 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.971450 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:33.973098 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.973079 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-dshm" pod "cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a" (UID: "cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:26:33.973190 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.973137 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-kube-api-access-7vlqz" pod "cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a" (UID: "cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a"). InnerVolumeSpecName "kube-api-access-7vlqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:33.973406 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.973391 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-tls-certs" pod "cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a" (UID: "cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:26:33.997642 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:33.997557 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4"] Oct 29 23:26:34.001000 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:34.000977 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a37090-fb7866b54-l42t4"] Oct 29 23:26:34.072675 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:34.072594 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vlqz\" (UniqueName: \"kubernetes.io/projected/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-kube-api-access-7vlqz\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:34.072675 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:34.072620 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:34.072675 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:34.072637 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:34.171297 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:34.171264 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49ae9910-d400-4cec-b171-d15541fa2789" path="/var/lib/kubelet/pods/49ae9910-d400-4cec-b171-d15541fa2789/volumes" Oct 29 23:26:34.284425 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:34.284394 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm"] Oct 29 23:26:34.288805 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:34.288780 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-dcacb9/test-model-dcacb9-kserve-5dfb8c5df9-s2tjm"] Oct 29 23:26:36.171071 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:36.171035 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a" path="/var/lib/kubelet/pods/cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a/volumes" Oct 29 23:26:42.991347 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:42.991263 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8"] Oct 29 23:26:42.991794 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:42.991558 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8" podUID="1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0" containerName="maas-api" containerID="cri-o://210cba46e83eb16c3f1e7861b0999c98d2b4141f505d681e297920a19c78e4c3" gracePeriod=30 Oct 29 23:26:43.056224 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:43.052751 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49"] Oct 29 23:26:43.499178 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:43.496413 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8" Oct 29 23:26:43.632147 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:43.632101 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-openshift-service-ca\") pod \"1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0\" (UID: \"1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0\") " Oct 29 23:26:43.632382 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:43.632316 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-maas-api-tls\") pod \"1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0\" (UID: \"1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0\") " Oct 29 23:26:43.632382 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:43.632361 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-kube-api-access-9d4jm\" (UniqueName: \"kubernetes.io/projected/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-kube-api-access-9d4jm\") pod \"1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0\" (UID: \"1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0\") " Oct 29 23:26:43.632748 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:43.632388 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-metrics-certs\") pod \"1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0\" (UID: \"1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0\") " Oct 29 23:26:43.647420 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:43.647377 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-openshift-service-ca" pod "1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0" (UID: "1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:26:43.647917 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:43.647882 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-metrics-certs" pod "1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0" (UID: "1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:26:43.653964 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:43.653933 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-maas-api-tls" pod "1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0" (UID: "1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:26:43.654303 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:43.654278 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-kube-api-access-9d4jm" pod "1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0" (UID: "1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0"). InnerVolumeSpecName "kube-api-access-9d4jm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:43.734893 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:43.734810 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:43.734893 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:43.734839 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d4jm\" (UniqueName: \"kubernetes.io/projected/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-kube-api-access-9d4jm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:43.734893 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:43.734849 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:43.734893 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:43.734857 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:44.025491 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:44.025404 4821 generic.go:356] "Generic (PLEG): container finished" podID="1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0" containerID="210cba46e83eb16c3f1e7861b0999c98d2b4141f505d681e297920a19c78e4c3" exitCode=0 Oct 29 23:26:44.025491 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:44.025460 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8" Oct 29 23:26:44.026014 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:44.025492 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8" event={"ID":"1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0","Type":"ContainerDied","Data":"210cba46e83eb16c3f1e7861b0999c98d2b4141f505d681e297920a19c78e4c3"} Oct 29 23:26:44.026014 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:44.025553 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8" event={"ID":"1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0","Type":"ContainerDied","Data":"7b954c702eae4c07c2ebe43b5ca3c2fb4ab2cf710f99a3ef22a222abaaf3b51b"} Oct 29 23:26:44.026014 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:44.025576 4821 scope.go:122] "RemoveContainer" containerID="210cba46e83eb16c3f1e7861b0999c98d2b4141f505d681e297920a19c78e4c3" Oct 29 23:26:44.038311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:44.038291 4821 scope.go:122] "RemoveContainer" containerID="210cba46e83eb16c3f1e7861b0999c98d2b4141f505d681e297920a19c78e4c3" Oct 29 23:26:44.038592 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:26:44.038574 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"210cba46e83eb16c3f1e7861b0999c98d2b4141f505d681e297920a19c78e4c3\": container with ID starting with 210cba46e83eb16c3f1e7861b0999c98d2b4141f505d681e297920a19c78e4c3 not found: ID does not exist" containerID="210cba46e83eb16c3f1e7861b0999c98d2b4141f505d681e297920a19c78e4c3" Oct 29 23:26:44.038653 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:44.038601 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"210cba46e83eb16c3f1e7861b0999c98d2b4141f505d681e297920a19c78e4c3"} err="failed to get container status \"210cba46e83eb16c3f1e7861b0999c98d2b4141f505d681e297920a19c78e4c3\": rpc error: code = NotFound desc = could not find container \"210cba46e83eb16c3f1e7861b0999c98d2b4141f505d681e297920a19c78e4c3\": container with ID starting with 210cba46e83eb16c3f1e7861b0999c98d2b4141f505d681e297920a19c78e4c3 not found: ID does not exist" Oct 29 23:26:44.151121 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:44.151089 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8"] Oct 29 23:26:44.202457 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:44.202400 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1a8107-7459d86d59-w5kv8"] Oct 29 23:26:46.172721 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:46.172687 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0" path="/var/lib/kubelet/pods/1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0/volumes" Oct 29 23:26:53.886739 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:26:53.886696 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:26:58.093961 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:58.093917 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" podUID="99a4a181-f0c4-438e-a5a3-f8764e83cfaf" containerName="main" containerID="cri-o://45fc73318a92e96e81e988c7723fcd2f1700c4fa05455a5bbd47c9d5d901e0f8" gracePeriod=45 Oct 29 23:26:58.100824 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:58.100793 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" podUID="99a4a181-f0c4-438e-a5a3-f8764e83cfaf" containerName="main" probeResult="failure" output="Get \"https://10.133.1.8:8000/ready\": dial tcp 10.133.1.8:8000: connect: connection refused" Oct 29 23:26:58.380894 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:58.380869 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:26:58.491863 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:58.491828 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-model-cache\") pod \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\" (UID: \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\") " Oct 29 23:26:58.492038 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:58.491873 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-dshm\") pod \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\" (UID: \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\") " Oct 29 23:26:58.492038 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:58.491944 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-kube-api-access-kf69t\" (UniqueName: \"kubernetes.io/projected/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-kube-api-access-kf69t\") pod \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\" (UID: \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\") " Oct 29 23:26:58.492038 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:58.492028 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-tmp-dir\") pod \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\" (UID: \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\") " Oct 29 23:26:58.492158 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:58.492101 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-tls-certs\" (UniqueName: \"kubernetes.io/secret/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-tls-certs\") pod \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\" (UID: \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\") " Oct 29 23:26:58.492158 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:58.492124 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-model-cache" pod "99a4a181-f0c4-438e-a5a3-f8764e83cfaf" (UID: "99a4a181-f0c4-438e-a5a3-f8764e83cfaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:26:58.492317 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:58.492300 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-home\" (UniqueName: \"kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-home\") pod \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\" (UID: \"99a4a181-f0c4-438e-a5a3-f8764e83cfaf\") " Oct 29 23:26:58.492426 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:58.492401 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-tmp-dir" pod "99a4a181-f0c4-438e-a5a3-f8764e83cfaf" (UID: "99a4a181-f0c4-438e-a5a3-f8764e83cfaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:26:58.492598 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:58.492576 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-home" pod "99a4a181-f0c4-438e-a5a3-f8764e83cfaf" (UID: "99a4a181-f0c4-438e-a5a3-f8764e83cfaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:26:58.493278 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:58.493247 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:58.493346 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:58.493278 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:58.493346 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:58.493292 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:58.494426 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:58.494403 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-dshm" pod "99a4a181-f0c4-438e-a5a3-f8764e83cfaf" (UID: "99a4a181-f0c4-438e-a5a3-f8764e83cfaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:26:58.494505 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:58.494475 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-kube-api-access-kf69t" pod "99a4a181-f0c4-438e-a5a3-f8764e83cfaf" (UID: "99a4a181-f0c4-438e-a5a3-f8764e83cfaf"). InnerVolumeSpecName "kube-api-access-kf69t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:26:58.494505 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:58.494485 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-tls-certs" pod "99a4a181-f0c4-438e-a5a3-f8764e83cfaf" (UID: "99a4a181-f0c4-438e-a5a3-f8764e83cfaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:26:58.594460 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:58.594423 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:58.594460 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:58.594453 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf69t\" (UniqueName: \"kubernetes.io/projected/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-kube-api-access-kf69t\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:58.594460 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:58.594462 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99a4a181-f0c4-438e-a5a3-f8764e83cfaf-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:26:59.183841 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:59.183803 4821 generic.go:356] "Generic (PLEG): container finished" podID="99a4a181-f0c4-438e-a5a3-f8764e83cfaf" containerID="45fc73318a92e96e81e988c7723fcd2f1700c4fa05455a5bbd47c9d5d901e0f8" exitCode=0 Oct 29 23:26:59.184432 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:59.183850 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" Oct 29 23:26:59.184432 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:59.183881 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" event={"ID":"99a4a181-f0c4-438e-a5a3-f8764e83cfaf","Type":"ContainerDied","Data":"45fc73318a92e96e81e988c7723fcd2f1700c4fa05455a5bbd47c9d5d901e0f8"} Oct 29 23:26:59.184432 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:59.183925 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49" event={"ID":"99a4a181-f0c4-438e-a5a3-f8764e83cfaf","Type":"ContainerDied","Data":"52e0e01fe455c84299ad08f93d5b5817abb7a0d26f1fd8c663a369bd34c9145d"} Oct 29 23:26:59.184432 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:59.183943 4821 scope.go:122] "RemoveContainer" containerID="45fc73318a92e96e81e988c7723fcd2f1700c4fa05455a5bbd47c9d5d901e0f8" Oct 29 23:26:59.195812 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:59.195791 4821 scope.go:122] "RemoveContainer" containerID="45fc73318a92e96e81e988c7723fcd2f1700c4fa05455a5bbd47c9d5d901e0f8" Oct 29 23:26:59.196088 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:26:59.196069 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45fc73318a92e96e81e988c7723fcd2f1700c4fa05455a5bbd47c9d5d901e0f8\": container with ID starting with 45fc73318a92e96e81e988c7723fcd2f1700c4fa05455a5bbd47c9d5d901e0f8 not found: ID does not exist" containerID="45fc73318a92e96e81e988c7723fcd2f1700c4fa05455a5bbd47c9d5d901e0f8" Oct 29 23:26:59.196152 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:59.196096 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45fc73318a92e96e81e988c7723fcd2f1700c4fa05455a5bbd47c9d5d901e0f8"} err="failed to get container status \"45fc73318a92e96e81e988c7723fcd2f1700c4fa05455a5bbd47c9d5d901e0f8\": rpc error: code = NotFound desc = could not find container \"45fc73318a92e96e81e988c7723fcd2f1700c4fa05455a5bbd47c9d5d901e0f8\": container with ID starting with 45fc73318a92e96e81e988c7723fcd2f1700c4fa05455a5bbd47c9d5d901e0f8 not found: ID does not exist" Oct 29 23:26:59.243796 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:59.243764 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49"] Oct 29 23:26:59.256806 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:26:59.256775 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-cc6328/test-model-cc6328-kserve-65d5d97477-h4b49"] Oct 29 23:27:00.170720 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:00.170687 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99a4a181-f0c4-438e-a5a3-f8764e83cfaf" path="/var/lib/kubelet/pods/99a4a181-f0c4-438e-a5a3-f8764e83cfaf/volumes" Oct 29 23:27:22.912789 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:22.912758 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c"] Oct 29 23:27:22.913388 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:22.912987 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c" podUID="78cb6576-8c0f-4a8b-9d05-72de5333c31d" containerName="maas-api" containerID="cri-o://2304ae03eb4b964b87a3bf119a55e83af5baa4547d6babbedc024a52a7e024a1" gracePeriod=30 Oct 29 23:27:23.530459 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:23.530411 4821 generic.go:356] "Generic (PLEG): container finished" podID="78cb6576-8c0f-4a8b-9d05-72de5333c31d" containerID="2304ae03eb4b964b87a3bf119a55e83af5baa4547d6babbedc024a52a7e024a1" exitCode=0 Oct 29 23:27:23.530641 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:23.530466 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c" event={"ID":"78cb6576-8c0f-4a8b-9d05-72de5333c31d","Type":"ContainerDied","Data":"2304ae03eb4b964b87a3bf119a55e83af5baa4547d6babbedc024a52a7e024a1"} Oct 29 23:27:23.557155 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:23.557131 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c" Oct 29 23:27:23.685717 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:23.685678 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78cb6576-8c0f-4a8b-9d05-72de5333c31d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78cb6576-8c0f-4a8b-9d05-72de5333c31d-maas-api-tls\") pod \"78cb6576-8c0f-4a8b-9d05-72de5333c31d\" (UID: \"78cb6576-8c0f-4a8b-9d05-72de5333c31d\") " Oct 29 23:27:23.685916 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:23.685843 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78cb6576-8c0f-4a8b-9d05-72de5333c31d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/78cb6576-8c0f-4a8b-9d05-72de5333c31d-metrics-certs\") pod \"78cb6576-8c0f-4a8b-9d05-72de5333c31d\" (UID: \"78cb6576-8c0f-4a8b-9d05-72de5333c31d\") " Oct 29 23:27:23.686041 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:23.686026 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78cb6576-8c0f-4a8b-9d05-72de5333c31d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78cb6576-8c0f-4a8b-9d05-72de5333c31d-openshift-service-ca\") pod \"78cb6576-8c0f-4a8b-9d05-72de5333c31d\" (UID: \"78cb6576-8c0f-4a8b-9d05-72de5333c31d\") " Oct 29 23:27:23.686107 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:23.686094 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78cb6576-8c0f-4a8b-9d05-72de5333c31d-kube-api-access-2ktww\" (UniqueName: \"kubernetes.io/projected/78cb6576-8c0f-4a8b-9d05-72de5333c31d-kube-api-access-2ktww\") pod \"78cb6576-8c0f-4a8b-9d05-72de5333c31d\" (UID: \"78cb6576-8c0f-4a8b-9d05-72de5333c31d\") " Oct 29 23:27:23.686489 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:23.686428 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78cb6576-8c0f-4a8b-9d05-72de5333c31d-openshift-service-ca" pod "78cb6576-8c0f-4a8b-9d05-72de5333c31d" (UID: "78cb6576-8c0f-4a8b-9d05-72de5333c31d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:23.686919 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:23.686896 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78cb6576-8c0f-4a8b-9d05-72de5333c31d-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:23.688607 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:23.688564 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78cb6576-8c0f-4a8b-9d05-72de5333c31d-maas-api-tls" pod "78cb6576-8c0f-4a8b-9d05-72de5333c31d" (UID: "78cb6576-8c0f-4a8b-9d05-72de5333c31d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:27:23.688795 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:23.688766 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78cb6576-8c0f-4a8b-9d05-72de5333c31d-metrics-certs" pod "78cb6576-8c0f-4a8b-9d05-72de5333c31d" (UID: "78cb6576-8c0f-4a8b-9d05-72de5333c31d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:27:23.688891 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:23.688812 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78cb6576-8c0f-4a8b-9d05-72de5333c31d-kube-api-access-2ktww" pod "78cb6576-8c0f-4a8b-9d05-72de5333c31d" (UID: "78cb6576-8c0f-4a8b-9d05-72de5333c31d"). InnerVolumeSpecName "kube-api-access-2ktww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:23.788372 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:23.788294 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ktww\" (UniqueName: \"kubernetes.io/projected/78cb6576-8c0f-4a8b-9d05-72de5333c31d-kube-api-access-2ktww\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:23.788372 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:23.788321 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78cb6576-8c0f-4a8b-9d05-72de5333c31d-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:23.788372 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:23.788331 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78cb6576-8c0f-4a8b-9d05-72de5333c31d-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:23.902955 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:23.902921 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8"] Oct 29 23:27:23.903137 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:23.903118 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" podUID="e438de1e-67a0-44a8-8ecd-e38ca050a541" containerName="istio-proxy" containerID="cri-o://cba975f80f91d84537dda3e737344dff853e8031ec6ccae24e3efe6f9df0e8b3" gracePeriod=30 Oct 29 23:27:23.903976 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:27:23.903940 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:27:24.538950 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:24.538870 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c" Oct 29 23:27:24.538950 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:24.538902 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c" event={"ID":"78cb6576-8c0f-4a8b-9d05-72de5333c31d","Type":"ContainerDied","Data":"237a0546b93cb8a395606561562db1106c5f8eb41e2eb4eba3bdebda0b7f2e01"} Oct 29 23:27:24.539598 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:24.538967 4821 scope.go:122] "RemoveContainer" containerID="2304ae03eb4b964b87a3bf119a55e83af5baa4547d6babbedc024a52a7e024a1" Oct 29 23:27:24.594432 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:24.594402 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c"] Oct 29 23:27:24.599501 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:24.599474 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dcacb9-695cdd8c7b-rtz5c"] Oct 29 23:27:26.172367 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:26.172333 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78cb6576-8c0f-4a8b-9d05-72de5333c31d" path="/var/lib/kubelet/pods/78cb6576-8c0f-4a8b-9d05-72de5333c31d/volumes" Oct 29 23:27:29.182570 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.182508 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 23:27:29.277424 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.277386 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-data\") pod \"e438de1e-67a0-44a8-8ecd-e38ca050a541\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " Oct 29 23:27:29.277632 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.277447 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-workload-socket\") pod \"e438de1e-67a0-44a8-8ecd-e38ca050a541\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " Oct 29 23:27:29.277632 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.277511 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e438de1e-67a0-44a8-8ecd-e38ca050a541-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e438de1e-67a0-44a8-8ecd-e38ca050a541-istiod-ca-cert\") pod \"e438de1e-67a0-44a8-8ecd-e38ca050a541\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " Oct 29 23:27:29.277632 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.277546 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-envoy\") pod \"e438de1e-67a0-44a8-8ecd-e38ca050a541\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " Oct 29 23:27:29.277632 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.277615 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-token\" (UniqueName: \"kubernetes.io/projected/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-token\") pod \"e438de1e-67a0-44a8-8ecd-e38ca050a541\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " Oct 29 23:27:29.277887 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.277644 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-workload-certs\") pod \"e438de1e-67a0-44a8-8ecd-e38ca050a541\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " Oct 29 23:27:29.277887 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.277704 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-workload-socket" pod "e438de1e-67a0-44a8-8ecd-e38ca050a541" (UID: "e438de1e-67a0-44a8-8ecd-e38ca050a541"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:29.277887 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.277725 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-podinfo\") pod \"e438de1e-67a0-44a8-8ecd-e38ca050a541\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " Oct 29 23:27:29.277887 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.277711 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-data" pod "e438de1e-67a0-44a8-8ecd-e38ca050a541" (UID: "e438de1e-67a0-44a8-8ecd-e38ca050a541"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:29.277887 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.277819 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e438de1e-67a0-44a8-8ecd-e38ca050a541-kube-api-access-27rj6\" (UniqueName: \"kubernetes.io/projected/e438de1e-67a0-44a8-8ecd-e38ca050a541-kube-api-access-27rj6\") pod \"e438de1e-67a0-44a8-8ecd-e38ca050a541\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " Oct 29 23:27:29.277887 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.277873 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-credential-socket\") pod \"e438de1e-67a0-44a8-8ecd-e38ca050a541\" (UID: \"e438de1e-67a0-44a8-8ecd-e38ca050a541\") " Oct 29 23:27:29.278638 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.278579 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-workload-certs" pod "e438de1e-67a0-44a8-8ecd-e38ca050a541" (UID: "e438de1e-67a0-44a8-8ecd-e38ca050a541"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:29.278911 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.278854 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-credential-socket" pod "e438de1e-67a0-44a8-8ecd-e38ca050a541" (UID: "e438de1e-67a0-44a8-8ecd-e38ca050a541"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:29.279264 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.279241 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e438de1e-67a0-44a8-8ecd-e38ca050a541-istiod-ca-cert" pod "e438de1e-67a0-44a8-8ecd-e38ca050a541" (UID: "e438de1e-67a0-44a8-8ecd-e38ca050a541"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:29.279541 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.279465 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:29.279541 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.279492 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:29.279541 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.279506 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:29.279541 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.279519 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:29.280997 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.280973 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-token" pod "e438de1e-67a0-44a8-8ecd-e38ca050a541" (UID: "e438de1e-67a0-44a8-8ecd-e38ca050a541"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:29.281483 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.281461 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-envoy" pod "e438de1e-67a0-44a8-8ecd-e38ca050a541" (UID: "e438de1e-67a0-44a8-8ecd-e38ca050a541"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:29.281600 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.281477 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-podinfo" pod "e438de1e-67a0-44a8-8ecd-e38ca050a541" (UID: "e438de1e-67a0-44a8-8ecd-e38ca050a541"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:27:29.283175 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.283151 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e438de1e-67a0-44a8-8ecd-e38ca050a541-kube-api-access-27rj6" pod "e438de1e-67a0-44a8-8ecd-e38ca050a541" (UID: "e438de1e-67a0-44a8-8ecd-e38ca050a541"). InnerVolumeSpecName "kube-api-access-27rj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:29.381262 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.381193 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e438de1e-67a0-44a8-8ecd-e38ca050a541-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:29.381262 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.381261 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:29.381262 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.381271 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:29.381535 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.381278 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e438de1e-67a0-44a8-8ecd-e38ca050a541-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:29.381535 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.381287 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27rj6\" (UniqueName: \"kubernetes.io/projected/e438de1e-67a0-44a8-8ecd-e38ca050a541-kube-api-access-27rj6\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:29.588728 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.588643 4821 generic.go:356] "Generic (PLEG): container finished" podID="e438de1e-67a0-44a8-8ecd-e38ca050a541" containerID="cba975f80f91d84537dda3e737344dff853e8031ec6ccae24e3efe6f9df0e8b3" exitCode=0 Oct 29 23:27:29.588728 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.588699 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" Oct 29 23:27:29.588920 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.588734 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" event={"ID":"e438de1e-67a0-44a8-8ecd-e38ca050a541","Type":"ContainerDied","Data":"cba975f80f91d84537dda3e737344dff853e8031ec6ccae24e3efe6f9df0e8b3"} Oct 29 23:27:29.588920 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.588796 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8" event={"ID":"e438de1e-67a0-44a8-8ecd-e38ca050a541","Type":"ContainerDied","Data":"0cb1439dc607e0e7e09baffb5c67715462030ff94d21eaa92e8a38d0cd39ec9f"} Oct 29 23:27:29.588920 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.588825 4821 scope.go:122] "RemoveContainer" containerID="cba975f80f91d84537dda3e737344dff853e8031ec6ccae24e3efe6f9df0e8b3" Oct 29 23:27:29.601082 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.601063 4821 scope.go:122] "RemoveContainer" containerID="cba975f80f91d84537dda3e737344dff853e8031ec6ccae24e3efe6f9df0e8b3" Oct 29 23:27:29.601363 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:27:29.601344 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cba975f80f91d84537dda3e737344dff853e8031ec6ccae24e3efe6f9df0e8b3\": container with ID starting with cba975f80f91d84537dda3e737344dff853e8031ec6ccae24e3efe6f9df0e8b3 not found: ID does not exist" containerID="cba975f80f91d84537dda3e737344dff853e8031ec6ccae24e3efe6f9df0e8b3" Oct 29 23:27:29.601455 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.601370 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cba975f80f91d84537dda3e737344dff853e8031ec6ccae24e3efe6f9df0e8b3"} err="failed to get container status \"cba975f80f91d84537dda3e737344dff853e8031ec6ccae24e3efe6f9df0e8b3\": rpc error: code = NotFound desc = could not find container \"cba975f80f91d84537dda3e737344dff853e8031ec6ccae24e3efe6f9df0e8b3\": container with ID starting with cba975f80f91d84537dda3e737344dff853e8031ec6ccae24e3efe6f9df0e8b3 not found: ID does not exist" Oct 29 23:27:29.644992 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.644965 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8"] Oct 29 23:27:29.666918 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:29.666836 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-575d59-openshift-default-6c9767b9bc-hztm8"] Oct 29 23:27:30.173823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:30.173777 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e438de1e-67a0-44a8-8ecd-e38ca050a541" path="/var/lib/kubelet/pods/e438de1e-67a0-44a8-8ecd-e38ca050a541/volumes" Oct 29 23:27:31.384003 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:31.383968 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd"] Oct 29 23:27:31.384581 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:31.384235 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" podUID="d560c3fe-b142-4ae7-a7da-c86260287c18" containerName="istio-proxy" containerID="cri-o://ff13156500ef13b127a38a578d875fa9b1c88774d6b281bd03f7939afc9be6f6" gracePeriod=30 Oct 29 23:27:32.964288 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:32.964256 4821 patch_prober.go:29] interesting pod/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:27:32.964684 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:32.964323 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" podUID="d560c3fe-b142-4ae7-a7da-c86260287c18" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:27:36.661803 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.661775 4821 generic.go:356] "Generic (PLEG): container finished" podID="d560c3fe-b142-4ae7-a7da-c86260287c18" containerID="ff13156500ef13b127a38a578d875fa9b1c88774d6b281bd03f7939afc9be6f6" exitCode=0 Oct 29 23:27:36.662213 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.661848 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" event={"ID":"d560c3fe-b142-4ae7-a7da-c86260287c18","Type":"ContainerDied","Data":"ff13156500ef13b127a38a578d875fa9b1c88774d6b281bd03f7939afc9be6f6"} Oct 29 23:27:36.662213 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.661910 4821 scope.go:122] "RemoveContainer" containerID="9c9a20c330af433a30b361d363749ffd131ec1fd73f3cd04bfe058a5c83d6199" Oct 29 23:27:36.701805 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.701774 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:27:36.797524 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.797484 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-envoy\") pod \"d560c3fe-b142-4ae7-a7da-c86260287c18\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " Oct 29 23:27:36.797732 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.797539 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-podinfo\") pod \"d560c3fe-b142-4ae7-a7da-c86260287c18\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " Oct 29 23:27:36.797732 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.797639 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-workload-socket\") pod \"d560c3fe-b142-4ae7-a7da-c86260287c18\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " Oct 29 23:27:36.797732 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.797682 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-token\" (UniqueName: \"kubernetes.io/projected/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-token\") pod \"d560c3fe-b142-4ae7-a7da-c86260287c18\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " Oct 29 23:27:36.797908 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.797754 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-data\") pod \"d560c3fe-b142-4ae7-a7da-c86260287c18\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " Oct 29 23:27:36.797908 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.797840 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d560c3fe-b142-4ae7-a7da-c86260287c18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d560c3fe-b142-4ae7-a7da-c86260287c18-istiod-ca-cert\") pod \"d560c3fe-b142-4ae7-a7da-c86260287c18\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " Oct 29 23:27:36.798021 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.797904 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d560c3fe-b142-4ae7-a7da-c86260287c18-kube-api-access-dk72l\" (UniqueName: \"kubernetes.io/projected/d560c3fe-b142-4ae7-a7da-c86260287c18-kube-api-access-dk72l\") pod \"d560c3fe-b142-4ae7-a7da-c86260287c18\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " Oct 29 23:27:36.798021 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.798010 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-workload-certs\") pod \"d560c3fe-b142-4ae7-a7da-c86260287c18\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " Oct 29 23:27:36.798178 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.798062 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-credential-socket\") pod \"d560c3fe-b142-4ae7-a7da-c86260287c18\" (UID: \"d560c3fe-b142-4ae7-a7da-c86260287c18\") " Oct 29 23:27:36.798178 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.798090 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-data" pod "d560c3fe-b142-4ae7-a7da-c86260287c18" (UID: "d560c3fe-b142-4ae7-a7da-c86260287c18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:36.799319 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.798334 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-workload-certs" pod "d560c3fe-b142-4ae7-a7da-c86260287c18" (UID: "d560c3fe-b142-4ae7-a7da-c86260287c18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:36.799319 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.798573 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-credential-socket" pod "d560c3fe-b142-4ae7-a7da-c86260287c18" (UID: "d560c3fe-b142-4ae7-a7da-c86260287c18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:36.799319 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.798760 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d560c3fe-b142-4ae7-a7da-c86260287c18-istiod-ca-cert" pod "d560c3fe-b142-4ae7-a7da-c86260287c18" (UID: "d560c3fe-b142-4ae7-a7da-c86260287c18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:36.799319 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.799165 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:36.799319 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.799187 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d560c3fe-b142-4ae7-a7da-c86260287c18-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:36.799319 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.799220 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:36.799319 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.799233 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:36.799879 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.799346 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-workload-socket" pod "d560c3fe-b142-4ae7-a7da-c86260287c18" (UID: "d560c3fe-b142-4ae7-a7da-c86260287c18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:36.802321 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.801468 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-envoy" pod "d560c3fe-b142-4ae7-a7da-c86260287c18" (UID: "d560c3fe-b142-4ae7-a7da-c86260287c18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:36.802321 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.801478 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-podinfo" pod "d560c3fe-b142-4ae7-a7da-c86260287c18" (UID: "d560c3fe-b142-4ae7-a7da-c86260287c18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:27:36.802321 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.801570 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-token" pod "d560c3fe-b142-4ae7-a7da-c86260287c18" (UID: "d560c3fe-b142-4ae7-a7da-c86260287c18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:36.802321 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.802167 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d560c3fe-b142-4ae7-a7da-c86260287c18-kube-api-access-dk72l" pod "d560c3fe-b142-4ae7-a7da-c86260287c18" (UID: "d560c3fe-b142-4ae7-a7da-c86260287c18"). InnerVolumeSpecName "kube-api-access-dk72l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:36.901475 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.901423 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk72l\" (UniqueName: \"kubernetes.io/projected/d560c3fe-b142-4ae7-a7da-c86260287c18-kube-api-access-dk72l\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:36.901475 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.901474 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:36.901475 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.901490 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:36.901773 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.901502 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d560c3fe-b142-4ae7-a7da-c86260287c18-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:36.901773 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:36.901516 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d560c3fe-b142-4ae7-a7da-c86260287c18-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:37.670096 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:37.670069 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" Oct 29 23:27:37.670096 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:37.670085 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd" event={"ID":"d560c3fe-b142-4ae7-a7da-c86260287c18","Type":"ContainerDied","Data":"c31dba572c3a649d70073d9bb9f0769931652f36705c9d3ecd411e6e23a926ec"} Oct 29 23:27:37.670631 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:37.670137 4821 scope.go:122] "RemoveContainer" containerID="ff13156500ef13b127a38a578d875fa9b1c88774d6b281bd03f7939afc9be6f6" Oct 29 23:27:37.779231 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:37.778379 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd"] Oct 29 23:27:37.795320 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:37.795280 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-dcacb9-openshift-default-54b777975-bqgzd"] Oct 29 23:27:38.171593 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:38.171560 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d560c3fe-b142-4ae7-a7da-c86260287c18" path="/var/lib/kubelet/pods/d560c3fe-b142-4ae7-a7da-c86260287c18/volumes" Oct 29 23:27:45.130220 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:45.126192 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq"] Oct 29 23:27:45.130220 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:45.126467 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq" podUID="e5c6effb-9b40-4c4f-aaa7-25bb59ce5469" containerName="maas-api" containerID="cri-o://5444c7447557639fe9d79044cf57751ff48bf3dfb52b8e0e6e6bfae2524c2758" gracePeriod=30 Oct 29 23:27:45.493237 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:45.493162 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq" Oct 29 23:27:45.543821 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:45.543786 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-openshift-service-ca\") pod \"e5c6effb-9b40-4c4f-aaa7-25bb59ce5469\" (UID: \"e5c6effb-9b40-4c4f-aaa7-25bb59ce5469\") " Oct 29 23:27:45.544017 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:45.543836 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-metrics-certs\") pod \"e5c6effb-9b40-4c4f-aaa7-25bb59ce5469\" (UID: \"e5c6effb-9b40-4c4f-aaa7-25bb59ce5469\") " Oct 29 23:27:45.544086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:45.544072 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-maas-api-tls\") pod \"e5c6effb-9b40-4c4f-aaa7-25bb59ce5469\" (UID: \"e5c6effb-9b40-4c4f-aaa7-25bb59ce5469\") " Oct 29 23:27:45.544158 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:45.544122 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-openshift-service-ca" pod "e5c6effb-9b40-4c4f-aaa7-25bb59ce5469" (UID: "e5c6effb-9b40-4c4f-aaa7-25bb59ce5469"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:45.544261 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:45.544234 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-kube-api-access-v49qm\" (UniqueName: \"kubernetes.io/projected/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-kube-api-access-v49qm\") pod \"e5c6effb-9b40-4c4f-aaa7-25bb59ce5469\" (UID: \"e5c6effb-9b40-4c4f-aaa7-25bb59ce5469\") " Oct 29 23:27:45.544789 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:45.544762 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:45.546629 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:45.546603 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-metrics-certs" pod "e5c6effb-9b40-4c4f-aaa7-25bb59ce5469" (UID: "e5c6effb-9b40-4c4f-aaa7-25bb59ce5469"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:27:45.546854 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:45.546826 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-maas-api-tls" pod "e5c6effb-9b40-4c4f-aaa7-25bb59ce5469" (UID: "e5c6effb-9b40-4c4f-aaa7-25bb59ce5469"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:27:45.547534 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:45.547507 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-kube-api-access-v49qm" pod "e5c6effb-9b40-4c4f-aaa7-25bb59ce5469" (UID: "e5c6effb-9b40-4c4f-aaa7-25bb59ce5469"). InnerVolumeSpecName "kube-api-access-v49qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:45.646261 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:45.646129 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:45.646261 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:45.646158 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v49qm\" (UniqueName: \"kubernetes.io/projected/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-kube-api-access-v49qm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:45.646261 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:45.646169 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:45.778240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:45.777734 4821 generic.go:356] "Generic (PLEG): container finished" podID="e5c6effb-9b40-4c4f-aaa7-25bb59ce5469" containerID="5444c7447557639fe9d79044cf57751ff48bf3dfb52b8e0e6e6bfae2524c2758" exitCode=0 Oct 29 23:27:45.778240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:45.777939 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq" event={"ID":"e5c6effb-9b40-4c4f-aaa7-25bb59ce5469","Type":"ContainerDied","Data":"5444c7447557639fe9d79044cf57751ff48bf3dfb52b8e0e6e6bfae2524c2758"} Oct 29 23:27:45.778240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:45.777997 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq" event={"ID":"e5c6effb-9b40-4c4f-aaa7-25bb59ce5469","Type":"ContainerDied","Data":"6153977aaea403cb3e07f8a6ee7ef104a20cfa68207da8a2d3884ab2cfa0fe9f"} Oct 29 23:27:45.778240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:45.778027 4821 scope.go:122] "RemoveContainer" containerID="5444c7447557639fe9d79044cf57751ff48bf3dfb52b8e0e6e6bfae2524c2758" Oct 29 23:27:45.778240 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:45.778230 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq" Oct 29 23:27:45.825032 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:45.825009 4821 scope.go:122] "RemoveContainer" containerID="5444c7447557639fe9d79044cf57751ff48bf3dfb52b8e0e6e6bfae2524c2758" Oct 29 23:27:45.825459 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:27:45.825439 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5444c7447557639fe9d79044cf57751ff48bf3dfb52b8e0e6e6bfae2524c2758\": container with ID starting with 5444c7447557639fe9d79044cf57751ff48bf3dfb52b8e0e6e6bfae2524c2758 not found: ID does not exist" containerID="5444c7447557639fe9d79044cf57751ff48bf3dfb52b8e0e6e6bfae2524c2758" Oct 29 23:27:45.825552 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:45.825467 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5444c7447557639fe9d79044cf57751ff48bf3dfb52b8e0e6e6bfae2524c2758"} err="failed to get container status \"5444c7447557639fe9d79044cf57751ff48bf3dfb52b8e0e6e6bfae2524c2758\": rpc error: code = NotFound desc = could not find container \"5444c7447557639fe9d79044cf57751ff48bf3dfb52b8e0e6e6bfae2524c2758\": container with ID starting with 5444c7447557639fe9d79044cf57751ff48bf3dfb52b8e0e6e6bfae2524c2758 not found: ID does not exist" Oct 29 23:27:45.865047 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:45.865018 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq"] Oct 29 23:27:45.878870 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:45.878845 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-3e5b21-5978c7656-lkdxq"] Oct 29 23:27:46.172530 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:46.172491 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5c6effb-9b40-4c4f-aaa7-25bb59ce5469" path="/var/lib/kubelet/pods/e5c6effb-9b40-4c4f-aaa7-25bb59ce5469/volumes" Oct 29 23:27:53.892939 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:27:53.892900 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:27:54.554013 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:54.553973 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw"] Oct 29 23:27:54.554526 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:54.554482 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" podUID="b74847df-b26e-4c6b-8f86-b86b5282c004" containerName="istio-proxy" containerID="cri-o://5d21db15320c85c53f4b47f02ff6286548fd99cdd79c37ee979ec6cf97a18f77" gracePeriod=30 Oct 29 23:27:58.927713 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:58.927676 4821 patch_prober.go:29] interesting pod/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:27:58.928247 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:58.927745 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" podUID="b74847df-b26e-4c6b-8f86-b86b5282c004" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:27:59.827667 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.827602 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:27:59.897741 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.897704 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-data\") pod \"b74847df-b26e-4c6b-8f86-b86b5282c004\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " Oct 29 23:27:59.897895 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.897805 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-podinfo\") pod \"b74847df-b26e-4c6b-8f86-b86b5282c004\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " Oct 29 23:27:59.897895 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.897867 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-workload-socket\") pod \"b74847df-b26e-4c6b-8f86-b86b5282c004\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " Oct 29 23:27:59.898035 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.897935 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b74847df-b26e-4c6b-8f86-b86b5282c004-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74847df-b26e-4c6b-8f86-b86b5282c004-istiod-ca-cert\") pod \"b74847df-b26e-4c6b-8f86-b86b5282c004\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " Oct 29 23:27:59.898035 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.897994 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-data" pod "b74847df-b26e-4c6b-8f86-b86b5282c004" (UID: "b74847df-b26e-4c6b-8f86-b86b5282c004"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:59.898035 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.898027 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-token\" (UniqueName: \"kubernetes.io/projected/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-token\") pod \"b74847df-b26e-4c6b-8f86-b86b5282c004\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " Oct 29 23:27:59.898185 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.898089 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b74847df-b26e-4c6b-8f86-b86b5282c004-kube-api-access-78z9f\" (UniqueName: \"kubernetes.io/projected/b74847df-b26e-4c6b-8f86-b86b5282c004-kube-api-access-78z9f\") pod \"b74847df-b26e-4c6b-8f86-b86b5282c004\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " Oct 29 23:27:59.898185 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.898170 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-envoy\") pod \"b74847df-b26e-4c6b-8f86-b86b5282c004\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " Oct 29 23:27:59.898351 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.898255 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-credential-socket\") pod \"b74847df-b26e-4c6b-8f86-b86b5282c004\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " Oct 29 23:27:59.898351 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.898309 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-workload-certs\") pod \"b74847df-b26e-4c6b-8f86-b86b5282c004\" (UID: \"b74847df-b26e-4c6b-8f86-b86b5282c004\") " Oct 29 23:27:59.898518 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.898252 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-workload-socket" pod "b74847df-b26e-4c6b-8f86-b86b5282c004" (UID: "b74847df-b26e-4c6b-8f86-b86b5282c004"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:59.898630 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.898434 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b74847df-b26e-4c6b-8f86-b86b5282c004-istiod-ca-cert" pod "b74847df-b26e-4c6b-8f86-b86b5282c004" (UID: "b74847df-b26e-4c6b-8f86-b86b5282c004"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:27:59.898725 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.898705 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-credential-socket" pod "b74847df-b26e-4c6b-8f86-b86b5282c004" (UID: "b74847df-b26e-4c6b-8f86-b86b5282c004"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:59.898791 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.898723 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-workload-certs" pod "b74847df-b26e-4c6b-8f86-b86b5282c004" (UID: "b74847df-b26e-4c6b-8f86-b86b5282c004"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:59.899220 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.899176 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74847df-b26e-4c6b-8f86-b86b5282c004-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:59.899299 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.899240 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:59.899299 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.899255 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:59.899299 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.899268 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:59.899299 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.899280 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:27:59.900676 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.900653 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-podinfo" pod "b74847df-b26e-4c6b-8f86-b86b5282c004" (UID: "b74847df-b26e-4c6b-8f86-b86b5282c004"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:27:59.901164 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.901132 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-envoy" pod "b74847df-b26e-4c6b-8f86-b86b5282c004" (UID: "b74847df-b26e-4c6b-8f86-b86b5282c004"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:27:59.901431 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.901406 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-token" pod "b74847df-b26e-4c6b-8f86-b86b5282c004" (UID: "b74847df-b26e-4c6b-8f86-b86b5282c004"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:59.901536 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.901438 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74847df-b26e-4c6b-8f86-b86b5282c004-kube-api-access-78z9f" pod "b74847df-b26e-4c6b-8f86-b86b5282c004" (UID: "b74847df-b26e-4c6b-8f86-b86b5282c004"). InnerVolumeSpecName "kube-api-access-78z9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:27:59.999367 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.999328 4821 generic.go:356] "Generic (PLEG): container finished" podID="b74847df-b26e-4c6b-8f86-b86b5282c004" containerID="5d21db15320c85c53f4b47f02ff6286548fd99cdd79c37ee979ec6cf97a18f77" exitCode=0 Oct 29 23:27:59.999896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.999391 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" Oct 29 23:27:59.999896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.999449 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" event={"ID":"b74847df-b26e-4c6b-8f86-b86b5282c004","Type":"ContainerDied","Data":"5d21db15320c85c53f4b47f02ff6286548fd99cdd79c37ee979ec6cf97a18f77"} Oct 29 23:27:59.999896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.999519 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw" event={"ID":"b74847df-b26e-4c6b-8f86-b86b5282c004","Type":"ContainerDied","Data":"cc6eafa394643addd478306d1c2e1502129939e76105f64d7a2c3e05e9eb7c21"} Oct 29 23:27:59.999896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:27:59.999548 4821 scope.go:122] "RemoveContainer" containerID="5d21db15320c85c53f4b47f02ff6286548fd99cdd79c37ee979ec6cf97a18f77" Oct 29 23:28:00.000517 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:00.000497 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:00.000624 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:00.000522 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78z9f\" (UniqueName: \"kubernetes.io/projected/b74847df-b26e-4c6b-8f86-b86b5282c004-kube-api-access-78z9f\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:00.000624 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:00.000545 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:00.000624 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:00.000556 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74847df-b26e-4c6b-8f86-b86b5282c004-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:00.011993 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:00.011975 4821 scope.go:122] "RemoveContainer" containerID="3f0561ec234de9dda8290ebc9313b4559c76f24e3d91a44c7b6ba8ae61f87ae0" Oct 29 23:28:00.022383 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:00.022365 4821 scope.go:122] "RemoveContainer" containerID="5d21db15320c85c53f4b47f02ff6286548fd99cdd79c37ee979ec6cf97a18f77" Oct 29 23:28:00.022702 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:28:00.022682 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d21db15320c85c53f4b47f02ff6286548fd99cdd79c37ee979ec6cf97a18f77\": container with ID starting with 5d21db15320c85c53f4b47f02ff6286548fd99cdd79c37ee979ec6cf97a18f77 not found: ID does not exist" containerID="5d21db15320c85c53f4b47f02ff6286548fd99cdd79c37ee979ec6cf97a18f77" Oct 29 23:28:00.022764 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:00.022712 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d21db15320c85c53f4b47f02ff6286548fd99cdd79c37ee979ec6cf97a18f77"} err="failed to get container status \"5d21db15320c85c53f4b47f02ff6286548fd99cdd79c37ee979ec6cf97a18f77\": rpc error: code = NotFound desc = could not find container \"5d21db15320c85c53f4b47f02ff6286548fd99cdd79c37ee979ec6cf97a18f77\": container with ID starting with 5d21db15320c85c53f4b47f02ff6286548fd99cdd79c37ee979ec6cf97a18f77 not found: ID does not exist" Oct 29 23:28:00.022764 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:00.022731 4821 scope.go:122] "RemoveContainer" containerID="3f0561ec234de9dda8290ebc9313b4559c76f24e3d91a44c7b6ba8ae61f87ae0" Oct 29 23:28:00.022999 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:28:00.022980 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0561ec234de9dda8290ebc9313b4559c76f24e3d91a44c7b6ba8ae61f87ae0\": container with ID starting with 3f0561ec234de9dda8290ebc9313b4559c76f24e3d91a44c7b6ba8ae61f87ae0 not found: ID does not exist" containerID="3f0561ec234de9dda8290ebc9313b4559c76f24e3d91a44c7b6ba8ae61f87ae0" Oct 29 23:28:00.023086 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:00.023006 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0561ec234de9dda8290ebc9313b4559c76f24e3d91a44c7b6ba8ae61f87ae0"} err="failed to get container status \"3f0561ec234de9dda8290ebc9313b4559c76f24e3d91a44c7b6ba8ae61f87ae0\": rpc error: code = NotFound desc = could not find container \"3f0561ec234de9dda8290ebc9313b4559c76f24e3d91a44c7b6ba8ae61f87ae0\": container with ID starting with 3f0561ec234de9dda8290ebc9313b4559c76f24e3d91a44c7b6ba8ae61f87ae0 not found: ID does not exist" Oct 29 23:28:00.058524 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:00.058495 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw"] Oct 29 23:28:00.066723 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:00.066694 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-3e5b21-openshift-default-7c7987569f-2pbtw"] Oct 29 23:28:00.189606 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:00.189571 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b74847df-b26e-4c6b-8f86-b86b5282c004" path="/var/lib/kubelet/pods/b74847df-b26e-4c6b-8f86-b86b5282c004/volumes" Oct 29 23:28:10.385227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:10.364547 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr"] Oct 29 23:28:10.386526 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:10.386483 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" podUID="dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" containerName="istio-proxy" containerID="cri-o://f0547326d2511f281bd162c96e3a31ed32544670f6847e8215d9a871fbe822d2" gracePeriod=30 Oct 29 23:28:15.723829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.723794 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:28:15.883525 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.883494 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-credential-socket\") pod \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " Oct 29 23:28:15.883733 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.883542 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-workload-certs\") pod \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " Oct 29 23:28:15.883733 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.883592 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-kube-api-access-t9hxk\" (UniqueName: \"kubernetes.io/projected/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-kube-api-access-t9hxk\") pod \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " Oct 29 23:28:15.883733 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.883637 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-workload-socket\") pod \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " Oct 29 23:28:15.883733 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.883676 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istiod-ca-cert\") pod \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " Oct 29 23:28:15.883975 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.883734 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-envoy\") pod \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " Oct 29 23:28:15.883975 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.883805 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-podinfo\") pod \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " Oct 29 23:28:15.883975 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.883826 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-credential-socket" pod "dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" (UID: "dc080a2a-5786-4e10-adbc-9b47e3ea4e1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:15.883975 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.883881 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-token\" (UniqueName: \"kubernetes.io/projected/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-token\") pod \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " Oct 29 23:28:15.884223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.883947 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-workload-certs" pod "dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" (UID: "dc080a2a-5786-4e10-adbc-9b47e3ea4e1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:15.884223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.884005 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-data\") pod \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\" (UID: \"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b\") " Oct 29 23:28:15.884223 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.884022 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-workload-socket" pod "dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" (UID: "dc080a2a-5786-4e10-adbc-9b47e3ea4e1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:15.884418 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.884329 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-data" pod "dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" (UID: "dc080a2a-5786-4e10-adbc-9b47e3ea4e1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:15.884497 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.884477 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istiod-ca-cert" pod "dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" (UID: "dc080a2a-5786-4e10-adbc-9b47e3ea4e1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:28:15.884812 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.884790 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:15.884909 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.884820 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:15.884909 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.884837 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:15.884909 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.884850 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:15.884909 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.884863 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:15.886163 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.886140 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-envoy" pod "dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" (UID: "dc080a2a-5786-4e10-adbc-9b47e3ea4e1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:15.886330 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.886311 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-token" pod "dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" (UID: "dc080a2a-5786-4e10-adbc-9b47e3ea4e1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:15.886401 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.886387 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-podinfo" pod "dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" (UID: "dc080a2a-5786-4e10-adbc-9b47e3ea4e1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:28:15.886817 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.886794 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-kube-api-access-t9hxk" pod "dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" (UID: "dc080a2a-5786-4e10-adbc-9b47e3ea4e1b"). InnerVolumeSpecName "kube-api-access-t9hxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:15.986549 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.986515 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:15.986549 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.986543 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:15.986549 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.986551 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:15.986792 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:15.986560 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9hxk\" (UniqueName: \"kubernetes.io/projected/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b-kube-api-access-t9hxk\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:16.159317 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:16.159234 4821 generic.go:356] "Generic (PLEG): container finished" podID="dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" containerID="f0547326d2511f281bd162c96e3a31ed32544670f6847e8215d9a871fbe822d2" exitCode=0 Oct 29 23:28:16.159317 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:16.159277 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" event={"ID":"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b","Type":"ContainerDied","Data":"f0547326d2511f281bd162c96e3a31ed32544670f6847e8215d9a871fbe822d2"} Oct 29 23:28:16.159317 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:16.159292 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" Oct 29 23:28:16.159604 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:16.159350 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr" event={"ID":"dc080a2a-5786-4e10-adbc-9b47e3ea4e1b","Type":"ContainerDied","Data":"8866065111d36643624815c5577f75479e81b7eda2e19f53c2d464788c04aef0"} Oct 29 23:28:16.159604 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:16.159372 4821 scope.go:122] "RemoveContainer" containerID="f0547326d2511f281bd162c96e3a31ed32544670f6847e8215d9a871fbe822d2" Oct 29 23:28:16.171365 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:16.171343 4821 scope.go:122] "RemoveContainer" containerID="1af6baa0ceabce775e19465651a91c140ce7af5bae38245349547331269625f6" Oct 29 23:28:16.183052 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:16.183031 4821 scope.go:122] "RemoveContainer" containerID="f0547326d2511f281bd162c96e3a31ed32544670f6847e8215d9a871fbe822d2" Oct 29 23:28:16.183365 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:28:16.183337 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0547326d2511f281bd162c96e3a31ed32544670f6847e8215d9a871fbe822d2\": container with ID starting with f0547326d2511f281bd162c96e3a31ed32544670f6847e8215d9a871fbe822d2 not found: ID does not exist" containerID="f0547326d2511f281bd162c96e3a31ed32544670f6847e8215d9a871fbe822d2" Oct 29 23:28:16.183475 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:16.183376 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0547326d2511f281bd162c96e3a31ed32544670f6847e8215d9a871fbe822d2"} err="failed to get container status \"f0547326d2511f281bd162c96e3a31ed32544670f6847e8215d9a871fbe822d2\": rpc error: code = NotFound desc = could not find container \"f0547326d2511f281bd162c96e3a31ed32544670f6847e8215d9a871fbe822d2\": container with ID starting with f0547326d2511f281bd162c96e3a31ed32544670f6847e8215d9a871fbe822d2 not found: ID does not exist" Oct 29 23:28:16.183475 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:16.183406 4821 scope.go:122] "RemoveContainer" containerID="1af6baa0ceabce775e19465651a91c140ce7af5bae38245349547331269625f6" Oct 29 23:28:16.183688 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:28:16.183664 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af6baa0ceabce775e19465651a91c140ce7af5bae38245349547331269625f6\": container with ID starting with 1af6baa0ceabce775e19465651a91c140ce7af5bae38245349547331269625f6 not found: ID does not exist" containerID="1af6baa0ceabce775e19465651a91c140ce7af5bae38245349547331269625f6" Oct 29 23:28:16.183753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:16.183698 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af6baa0ceabce775e19465651a91c140ce7af5bae38245349547331269625f6"} err="failed to get container status \"1af6baa0ceabce775e19465651a91c140ce7af5bae38245349547331269625f6\": rpc error: code = NotFound desc = could not find container \"1af6baa0ceabce775e19465651a91c140ce7af5bae38245349547331269625f6\": container with ID starting with 1af6baa0ceabce775e19465651a91c140ce7af5bae38245349547331269625f6 not found: ID does not exist" Oct 29 23:28:16.224285 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:16.224192 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr"] Oct 29 23:28:16.228029 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:16.228012 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-c880c7-openshift-default-7fccd7bf79-4gtlr"] Oct 29 23:28:18.171726 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:18.171693 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" path="/var/lib/kubelet/pods/dc080a2a-5786-4e10-adbc-9b47e3ea4e1b/volumes" Oct 29 23:28:21.200045 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:21.200009 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w"] Oct 29 23:28:21.237255 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:21.237213 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl"] Oct 29 23:28:23.886496 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:28:23.886462 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:28:26.200885 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:26.200827 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w" podUID="153f58c3-7d4f-4aa1-b7a5-2f807a0b1811" containerName="payload-processing" containerID="cri-o://31e63dfd0f32f3353f7aac3bdf9f0d3f9818e8aa741b00920de6ef688bbc72a3" gracePeriod=25 Oct 29 23:28:26.238552 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:26.238500 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl" podUID="04c222af-dd88-48ed-88a0-ad27b981b410" containerName="payload-pre-processing" containerID="cri-o://35648b56f84d8cd533719f5598611af5aa077dab1987010e5bea4d17e0858ccf" gracePeriod=25 Oct 29 23:28:26.510444 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:26.510415 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w" Oct 29 23:28:26.602021 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:26.601994 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl" Oct 29 23:28:26.643528 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:26.643497 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04c222af-dd88-48ed-88a0-ad27b981b410-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04c222af-dd88-48ed-88a0-ad27b981b410-plugins-config-volume\") pod \"04c222af-dd88-48ed-88a0-ad27b981b410\" (UID: \"04c222af-dd88-48ed-88a0-ad27b981b410\") " Oct 29 23:28:26.643697 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:26.643596 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153f58c3-7d4f-4aa1-b7a5-2f807a0b1811-kube-api-access-d5656\" (UniqueName: \"kubernetes.io/projected/153f58c3-7d4f-4aa1-b7a5-2f807a0b1811-kube-api-access-d5656\") pod \"153f58c3-7d4f-4aa1-b7a5-2f807a0b1811\" (UID: \"153f58c3-7d4f-4aa1-b7a5-2f807a0b1811\") " Oct 29 23:28:26.643697 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:26.643667 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04c222af-dd88-48ed-88a0-ad27b981b410-kube-api-access-cbn66\" (UniqueName: \"kubernetes.io/projected/04c222af-dd88-48ed-88a0-ad27b981b410-kube-api-access-cbn66\") pod \"04c222af-dd88-48ed-88a0-ad27b981b410\" (UID: \"04c222af-dd88-48ed-88a0-ad27b981b410\") " Oct 29 23:28:26.643697 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:26.643690 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/153f58c3-7d4f-4aa1-b7a5-2f807a0b1811-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153f58c3-7d4f-4aa1-b7a5-2f807a0b1811-plugins-config-volume\") pod \"153f58c3-7d4f-4aa1-b7a5-2f807a0b1811\" (UID: \"153f58c3-7d4f-4aa1-b7a5-2f807a0b1811\") " Oct 29 23:28:26.644013 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:26.643941 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04c222af-dd88-48ed-88a0-ad27b981b410-plugins-config-volume" pod "04c222af-dd88-48ed-88a0-ad27b981b410" (UID: "04c222af-dd88-48ed-88a0-ad27b981b410"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:28:26.644134 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:26.644110 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/153f58c3-7d4f-4aa1-b7a5-2f807a0b1811-plugins-config-volume" pod "153f58c3-7d4f-4aa1-b7a5-2f807a0b1811" (UID: "153f58c3-7d4f-4aa1-b7a5-2f807a0b1811"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:28:26.644311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:26.644257 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153f58c3-7d4f-4aa1-b7a5-2f807a0b1811-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:26.644311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:26.644283 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04c222af-dd88-48ed-88a0-ad27b981b410-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:26.646619 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:26.646589 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153f58c3-7d4f-4aa1-b7a5-2f807a0b1811-kube-api-access-d5656" pod "153f58c3-7d4f-4aa1-b7a5-2f807a0b1811" (UID: "153f58c3-7d4f-4aa1-b7a5-2f807a0b1811"). InnerVolumeSpecName "kube-api-access-d5656". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:26.647059 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:26.647039 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c222af-dd88-48ed-88a0-ad27b981b410-kube-api-access-cbn66" pod "04c222af-dd88-48ed-88a0-ad27b981b410" (UID: "04c222af-dd88-48ed-88a0-ad27b981b410"). InnerVolumeSpecName "kube-api-access-cbn66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:26.745354 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:26.745269 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5656\" (UniqueName: \"kubernetes.io/projected/153f58c3-7d4f-4aa1-b7a5-2f807a0b1811-kube-api-access-d5656\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:26.745354 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:26.745300 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbn66\" (UniqueName: \"kubernetes.io/projected/04c222af-dd88-48ed-88a0-ad27b981b410-kube-api-access-cbn66\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:27.254189 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:27.254153 4821 generic.go:356] "Generic (PLEG): container finished" podID="04c222af-dd88-48ed-88a0-ad27b981b410" containerID="35648b56f84d8cd533719f5598611af5aa077dab1987010e5bea4d17e0858ccf" exitCode=0 Oct 29 23:28:27.254663 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:27.254223 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl" Oct 29 23:28:27.254663 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:27.254229 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl" event={"ID":"04c222af-dd88-48ed-88a0-ad27b981b410","Type":"ContainerDied","Data":"35648b56f84d8cd533719f5598611af5aa077dab1987010e5bea4d17e0858ccf"} Oct 29 23:28:27.254663 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:27.254280 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl" event={"ID":"04c222af-dd88-48ed-88a0-ad27b981b410","Type":"ContainerDied","Data":"1711df1630232cab6c760c2845d0c5bd15c1dbf386a80a37a60bac39b16a280e"} Oct 29 23:28:27.254663 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:27.254303 4821 scope.go:122] "RemoveContainer" containerID="35648b56f84d8cd533719f5598611af5aa077dab1987010e5bea4d17e0858ccf" Oct 29 23:28:27.255704 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:27.255670 4821 generic.go:356] "Generic (PLEG): container finished" podID="153f58c3-7d4f-4aa1-b7a5-2f807a0b1811" containerID="31e63dfd0f32f3353f7aac3bdf9f0d3f9818e8aa741b00920de6ef688bbc72a3" exitCode=0 Oct 29 23:28:27.255822 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:27.255710 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w" event={"ID":"153f58c3-7d4f-4aa1-b7a5-2f807a0b1811","Type":"ContainerDied","Data":"31e63dfd0f32f3353f7aac3bdf9f0d3f9818e8aa741b00920de6ef688bbc72a3"} Oct 29 23:28:27.255822 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:27.255725 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w" Oct 29 23:28:27.255822 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:27.255738 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w" event={"ID":"153f58c3-7d4f-4aa1-b7a5-2f807a0b1811","Type":"ContainerDied","Data":"b67916c139e5dff29ec6cbfbbdcf884828c8ce677e007066d4511bd3c81eb936"} Oct 29 23:28:27.266710 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:27.266694 4821 scope.go:122] "RemoveContainer" containerID="35648b56f84d8cd533719f5598611af5aa077dab1987010e5bea4d17e0858ccf" Oct 29 23:28:27.266964 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:28:27.266946 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35648b56f84d8cd533719f5598611af5aa077dab1987010e5bea4d17e0858ccf\": container with ID starting with 35648b56f84d8cd533719f5598611af5aa077dab1987010e5bea4d17e0858ccf not found: ID does not exist" containerID="35648b56f84d8cd533719f5598611af5aa077dab1987010e5bea4d17e0858ccf" Oct 29 23:28:27.267007 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:27.266972 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35648b56f84d8cd533719f5598611af5aa077dab1987010e5bea4d17e0858ccf"} err="failed to get container status \"35648b56f84d8cd533719f5598611af5aa077dab1987010e5bea4d17e0858ccf\": rpc error: code = NotFound desc = could not find container \"35648b56f84d8cd533719f5598611af5aa077dab1987010e5bea4d17e0858ccf\": container with ID starting with 35648b56f84d8cd533719f5598611af5aa077dab1987010e5bea4d17e0858ccf not found: ID does not exist" Oct 29 23:28:27.267007 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:27.266990 4821 scope.go:122] "RemoveContainer" containerID="31e63dfd0f32f3353f7aac3bdf9f0d3f9818e8aa741b00920de6ef688bbc72a3" Oct 29 23:28:27.276287 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:27.276267 4821 scope.go:122] "RemoveContainer" containerID="31e63dfd0f32f3353f7aac3bdf9f0d3f9818e8aa741b00920de6ef688bbc72a3" Oct 29 23:28:27.276558 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:28:27.276526 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e63dfd0f32f3353f7aac3bdf9f0d3f9818e8aa741b00920de6ef688bbc72a3\": container with ID starting with 31e63dfd0f32f3353f7aac3bdf9f0d3f9818e8aa741b00920de6ef688bbc72a3 not found: ID does not exist" containerID="31e63dfd0f32f3353f7aac3bdf9f0d3f9818e8aa741b00920de6ef688bbc72a3" Oct 29 23:28:27.276604 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:27.276565 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e63dfd0f32f3353f7aac3bdf9f0d3f9818e8aa741b00920de6ef688bbc72a3"} err="failed to get container status \"31e63dfd0f32f3353f7aac3bdf9f0d3f9818e8aa741b00920de6ef688bbc72a3\": rpc error: code = NotFound desc = could not find container \"31e63dfd0f32f3353f7aac3bdf9f0d3f9818e8aa741b00920de6ef688bbc72a3\": container with ID starting with 31e63dfd0f32f3353f7aac3bdf9f0d3f9818e8aa741b00920de6ef688bbc72a3 not found: ID does not exist" Oct 29 23:28:27.310950 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:27.310918 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl"] Oct 29 23:28:27.323292 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:27.323268 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-5e52e0-787cc96ff9-dcrfl"] Oct 29 23:28:27.348440 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:27.348409 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w"] Oct 29 23:28:27.352496 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:27.352473 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-5e52e0-6c4c95cd4d-xqt5w"] Oct 29 23:28:27.751041 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:27.751001 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj"] Oct 29 23:28:27.751272 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:27.751248 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" podUID="8d192d0e-6dcf-4979-a503-6047643e055e" containerName="maas-api" containerID="cri-o://5e5c8fe5676eed629d233f5fab6040542f0f9a013d5086a87cf512732f67096c" gracePeriod=30 Oct 29 23:28:28.306242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:28.305804 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04c222af-dd88-48ed-88a0-ad27b981b410" path="/var/lib/kubelet/pods/04c222af-dd88-48ed-88a0-ad27b981b410/volumes" Oct 29 23:28:28.310248 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:28.309930 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="153f58c3-7d4f-4aa1-b7a5-2f807a0b1811" path="/var/lib/kubelet/pods/153f58c3-7d4f-4aa1-b7a5-2f807a0b1811/volumes" Oct 29 23:28:28.355456 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:28.355404 4821 generic.go:356] "Generic (PLEG): container finished" podID="8d192d0e-6dcf-4979-a503-6047643e055e" containerID="5e5c8fe5676eed629d233f5fab6040542f0f9a013d5086a87cf512732f67096c" exitCode=0 Oct 29 23:28:28.355709 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:28.355513 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" event={"ID":"8d192d0e-6dcf-4979-a503-6047643e055e","Type":"ContainerDied","Data":"5e5c8fe5676eed629d233f5fab6040542f0f9a013d5086a87cf512732f67096c"} Oct 29 23:28:28.355709 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:28.355557 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" event={"ID":"8d192d0e-6dcf-4979-a503-6047643e055e","Type":"ContainerDied","Data":"90b315d661564fb89c31e65a328358abcfc34ff417ec8b7709c2110feaa0ea99"} Oct 29 23:28:28.355709 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:28.355581 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90b315d661564fb89c31e65a328358abcfc34ff417ec8b7709c2110feaa0ea99" Oct 29 23:28:28.355709 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:28.355645 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" Oct 29 23:28:28.440431 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:28.440372 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d192d0e-6dcf-4979-a503-6047643e055e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d192d0e-6dcf-4979-a503-6047643e055e-openshift-service-ca\") pod \"8d192d0e-6dcf-4979-a503-6047643e055e\" (UID: \"8d192d0e-6dcf-4979-a503-6047643e055e\") " Oct 29 23:28:28.440702 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:28.440460 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d192d0e-6dcf-4979-a503-6047643e055e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d192d0e-6dcf-4979-a503-6047643e055e-maas-api-tls\") pod \"8d192d0e-6dcf-4979-a503-6047643e055e\" (UID: \"8d192d0e-6dcf-4979-a503-6047643e055e\") " Oct 29 23:28:28.440702 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:28.440646 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d192d0e-6dcf-4979-a503-6047643e055e-kube-api-access-85d44\" (UniqueName: \"kubernetes.io/projected/8d192d0e-6dcf-4979-a503-6047643e055e-kube-api-access-85d44\") pod \"8d192d0e-6dcf-4979-a503-6047643e055e\" (UID: \"8d192d0e-6dcf-4979-a503-6047643e055e\") " Oct 29 23:28:28.440702 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:28.440689 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d192d0e-6dcf-4979-a503-6047643e055e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d192d0e-6dcf-4979-a503-6047643e055e-metrics-certs\") pod \"8d192d0e-6dcf-4979-a503-6047643e055e\" (UID: \"8d192d0e-6dcf-4979-a503-6047643e055e\") " Oct 29 23:28:28.449124 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:28.449093 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d192d0e-6dcf-4979-a503-6047643e055e-openshift-service-ca" pod "8d192d0e-6dcf-4979-a503-6047643e055e" (UID: "8d192d0e-6dcf-4979-a503-6047643e055e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:28:28.458978 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:28.458930 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d192d0e-6dcf-4979-a503-6047643e055e-maas-api-tls" pod "8d192d0e-6dcf-4979-a503-6047643e055e" (UID: "8d192d0e-6dcf-4979-a503-6047643e055e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:28:28.459695 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:28.459482 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d192d0e-6dcf-4979-a503-6047643e055e-kube-api-access-85d44" pod "8d192d0e-6dcf-4979-a503-6047643e055e" (UID: "8d192d0e-6dcf-4979-a503-6047643e055e"). InnerVolumeSpecName "kube-api-access-85d44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:28.460195 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:28.460006 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d192d0e-6dcf-4979-a503-6047643e055e-metrics-certs" pod "8d192d0e-6dcf-4979-a503-6047643e055e" (UID: "8d192d0e-6dcf-4979-a503-6047643e055e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:28:28.542682 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:28.542585 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d192d0e-6dcf-4979-a503-6047643e055e-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:28.542682 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:28.542644 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d192d0e-6dcf-4979-a503-6047643e055e-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:28.542682 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:28.542659 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85d44\" (UniqueName: \"kubernetes.io/projected/8d192d0e-6dcf-4979-a503-6047643e055e-kube-api-access-85d44\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:28.542682 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:28.542674 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d192d0e-6dcf-4979-a503-6047643e055e-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:29.362357 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:29.362325 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj" Oct 29 23:28:29.418162 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:29.418129 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj"] Oct 29 23:28:29.430902 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:29.430879 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-755453-7cf678ddc5-4d5cj"] Oct 29 23:28:30.170233 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:30.170183 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d192d0e-6dcf-4979-a503-6047643e055e" path="/var/lib/kubelet/pods/8d192d0e-6dcf-4979-a503-6047643e055e/volumes" Oct 29 23:28:30.792986 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:30.792952 4821 scope.go:122] "RemoveContainer" containerID="5e5c8fe5676eed629d233f5fab6040542f0f9a013d5086a87cf512732f67096c" Oct 29 23:28:40.745427 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:40.745393 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc"] Oct 29 23:28:40.745890 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:40.745641 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc" podUID="7d7c0ca7-37fe-4a65-a09b-e127842adb0b" containerName="maas-api" containerID="cri-o://825eb47b71364548d39015ee48f5fc79f2003bc6690339f240fc06c8b3b00cff" gracePeriod=30 Oct 29 23:28:41.150222 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:41.145851 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc" Oct 29 23:28:41.254270 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:41.254232 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-maas-api-tls\") pod \"7d7c0ca7-37fe-4a65-a09b-e127842adb0b\" (UID: \"7d7c0ca7-37fe-4a65-a09b-e127842adb0b\") " Oct 29 23:28:41.254802 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:41.254775 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-openshift-service-ca\") pod \"7d7c0ca7-37fe-4a65-a09b-e127842adb0b\" (UID: \"7d7c0ca7-37fe-4a65-a09b-e127842adb0b\") " Oct 29 23:28:41.255429 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:41.255314 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-openshift-service-ca" pod "7d7c0ca7-37fe-4a65-a09b-e127842adb0b" (UID: "7d7c0ca7-37fe-4a65-a09b-e127842adb0b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:28:41.255687 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:41.255670 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-kube-api-access-n2hwx\" (UniqueName: \"kubernetes.io/projected/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-kube-api-access-n2hwx\") pod \"7d7c0ca7-37fe-4a65-a09b-e127842adb0b\" (UID: \"7d7c0ca7-37fe-4a65-a09b-e127842adb0b\") " Oct 29 23:28:41.256307 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:41.256285 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-metrics-certs\") pod \"7d7c0ca7-37fe-4a65-a09b-e127842adb0b\" (UID: \"7d7c0ca7-37fe-4a65-a09b-e127842adb0b\") " Oct 29 23:28:41.257428 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:41.257405 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:41.270747 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:41.269867 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-maas-api-tls" pod "7d7c0ca7-37fe-4a65-a09b-e127842adb0b" (UID: "7d7c0ca7-37fe-4a65-a09b-e127842adb0b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:28:41.270747 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:41.270478 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-metrics-certs" pod "7d7c0ca7-37fe-4a65-a09b-e127842adb0b" (UID: "7d7c0ca7-37fe-4a65-a09b-e127842adb0b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:28:41.274571 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:41.271673 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-kube-api-access-n2hwx" pod "7d7c0ca7-37fe-4a65-a09b-e127842adb0b" (UID: "7d7c0ca7-37fe-4a65-a09b-e127842adb0b"). InnerVolumeSpecName "kube-api-access-n2hwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:41.358892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:41.358863 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:41.359061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:41.358896 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2hwx\" (UniqueName: \"kubernetes.io/projected/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-kube-api-access-n2hwx\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:41.359061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:41.358907 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d7c0ca7-37fe-4a65-a09b-e127842adb0b-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:41.541574 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:41.541536 4821 generic.go:356] "Generic (PLEG): container finished" podID="7d7c0ca7-37fe-4a65-a09b-e127842adb0b" containerID="825eb47b71364548d39015ee48f5fc79f2003bc6690339f240fc06c8b3b00cff" exitCode=0 Oct 29 23:28:41.541748 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:41.541591 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc" Oct 29 23:28:41.541748 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:41.541621 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc" event={"ID":"7d7c0ca7-37fe-4a65-a09b-e127842adb0b","Type":"ContainerDied","Data":"825eb47b71364548d39015ee48f5fc79f2003bc6690339f240fc06c8b3b00cff"} Oct 29 23:28:41.541748 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:41.541680 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc" event={"ID":"7d7c0ca7-37fe-4a65-a09b-e127842adb0b","Type":"ContainerDied","Data":"91166794adca5b90605ef6c4c27593be76a020795458dd9d77ea0ee17035c803"} Oct 29 23:28:41.541748 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:41.541712 4821 scope.go:122] "RemoveContainer" containerID="825eb47b71364548d39015ee48f5fc79f2003bc6690339f240fc06c8b3b00cff" Oct 29 23:28:41.552522 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:41.552502 4821 scope.go:122] "RemoveContainer" containerID="825eb47b71364548d39015ee48f5fc79f2003bc6690339f240fc06c8b3b00cff" Oct 29 23:28:41.552791 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:28:41.552771 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"825eb47b71364548d39015ee48f5fc79f2003bc6690339f240fc06c8b3b00cff\": container with ID starting with 825eb47b71364548d39015ee48f5fc79f2003bc6690339f240fc06c8b3b00cff not found: ID does not exist" containerID="825eb47b71364548d39015ee48f5fc79f2003bc6690339f240fc06c8b3b00cff" Oct 29 23:28:41.552879 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:41.552802 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"825eb47b71364548d39015ee48f5fc79f2003bc6690339f240fc06c8b3b00cff"} err="failed to get container status \"825eb47b71364548d39015ee48f5fc79f2003bc6690339f240fc06c8b3b00cff\": rpc error: code = NotFound desc = could not find container \"825eb47b71364548d39015ee48f5fc79f2003bc6690339f240fc06c8b3b00cff\": container with ID starting with 825eb47b71364548d39015ee48f5fc79f2003bc6690339f240fc06c8b3b00cff not found: ID does not exist" Oct 29 23:28:41.597324 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:41.597293 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc"] Oct 29 23:28:41.601266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:41.601242 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-5e52e0-6978cd9d68-hvcsc"] Oct 29 23:28:42.175577 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:42.175534 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d7c0ca7-37fe-4a65-a09b-e127842adb0b" path="/var/lib/kubelet/pods/7d7c0ca7-37fe-4a65-a09b-e127842adb0b/volumes" Oct 29 23:28:42.396696 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:42.396659 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7"] Oct 29 23:28:42.419545 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:42.419500 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n"] Oct 29 23:28:43.965615 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:43.965581 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b"] Oct 29 23:28:43.997729 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:43.997684 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q"] Oct 29 23:28:44.000940 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:44.000910 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk"] Oct 29 23:28:53.887743 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:28:53.887705 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:28:54.895773 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:54.895731 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf"] Oct 29 23:28:54.908936 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:54.908909 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr"] Oct 29 23:28:54.919728 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:54.919700 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4"] Oct 29 23:28:57.418354 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.418292 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" containerID="cri-o://6098d9779a9f49651f7131fbe7ced8656360875db4c1c11387a3d30af7175bce" gracePeriod=45 Oct 29 23:28:57.440433 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.440401 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" containerID="cri-o://f7593facdec14c6344cbe725bc09100f14ab15f7b00d27f159d3dfc51147e91d" gracePeriod=45 Oct 29 23:28:57.655695 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.655494 4821 generic.go:356] "Generic (PLEG): container finished" podID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerID="6098d9779a9f49651f7131fbe7ced8656360875db4c1c11387a3d30af7175bce" exitCode=0 Oct 29 23:28:57.655695 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.655641 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" event={"ID":"1231f78b-aaa2-4595-a11e-fcdbf984521e","Type":"ContainerDied","Data":"6098d9779a9f49651f7131fbe7ced8656360875db4c1c11387a3d30af7175bce"} Oct 29 23:28:57.658099 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.658070 4821 generic.go:356] "Generic (PLEG): container finished" podID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerID="f7593facdec14c6344cbe725bc09100f14ab15f7b00d27f159d3dfc51147e91d" exitCode=0 Oct 29 23:28:57.658278 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.658139 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" event={"ID":"e23010a6-7ea5-40e3-b67d-f88d04c30e81","Type":"ContainerDied","Data":"f7593facdec14c6344cbe725bc09100f14ab15f7b00d27f159d3dfc51147e91d"} Oct 29 23:28:57.729680 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.729651 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 23:28:57.732968 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.732943 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 23:28:57.840302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.840272 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-dshm\") pod \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\" (UID: \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\") " Oct 29 23:28:57.840488 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.840314 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-dshm\") pod \"1231f78b-aaa2-4595-a11e-fcdbf984521e\" (UID: \"1231f78b-aaa2-4595-a11e-fcdbf984521e\") " Oct 29 23:28:57.840488 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.840345 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e23010a6-7ea5-40e3-b67d-f88d04c30e81-kube-api-access-n9g6v\" (UniqueName: \"kubernetes.io/projected/e23010a6-7ea5-40e3-b67d-f88d04c30e81-kube-api-access-n9g6v\") pod \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\" (UID: \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\") " Oct 29 23:28:57.840488 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.840403 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-model-cache\") pod \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\" (UID: \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\") " Oct 29 23:28:57.840488 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.840433 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-home\" (UniqueName: \"kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-home\") pod \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\" (UID: \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\") " Oct 29 23:28:57.840729 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.840501 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-home\" (UniqueName: \"kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-home\") pod \"1231f78b-aaa2-4595-a11e-fcdbf984521e\" (UID: \"1231f78b-aaa2-4595-a11e-fcdbf984521e\") " Oct 29 23:28:57.840729 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.840555 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-tmp-dir\") pod \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\" (UID: \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\") " Oct 29 23:28:57.840729 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.840618 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-tmp-dir\") pod \"1231f78b-aaa2-4595-a11e-fcdbf984521e\" (UID: \"1231f78b-aaa2-4595-a11e-fcdbf984521e\") " Oct 29 23:28:57.840729 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.840698 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e23010a6-7ea5-40e3-b67d-f88d04c30e81-tls-certs\" (UniqueName: \"kubernetes.io/secret/e23010a6-7ea5-40e3-b67d-f88d04c30e81-tls-certs\") pod \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\" (UID: \"e23010a6-7ea5-40e3-b67d-f88d04c30e81\") " Oct 29 23:28:57.840930 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.840803 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1231f78b-aaa2-4595-a11e-fcdbf984521e-tls-certs\" (UniqueName: \"kubernetes.io/secret/1231f78b-aaa2-4595-a11e-fcdbf984521e-tls-certs\") pod \"1231f78b-aaa2-4595-a11e-fcdbf984521e\" (UID: \"1231f78b-aaa2-4595-a11e-fcdbf984521e\") " Oct 29 23:28:57.840930 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.840850 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1231f78b-aaa2-4595-a11e-fcdbf984521e-kube-api-access-cszm2\" (UniqueName: \"kubernetes.io/projected/1231f78b-aaa2-4595-a11e-fcdbf984521e-kube-api-access-cszm2\") pod \"1231f78b-aaa2-4595-a11e-fcdbf984521e\" (UID: \"1231f78b-aaa2-4595-a11e-fcdbf984521e\") " Oct 29 23:28:57.840930 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.840894 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-model-cache\") pod \"1231f78b-aaa2-4595-a11e-fcdbf984521e\" (UID: \"1231f78b-aaa2-4595-a11e-fcdbf984521e\") " Oct 29 23:28:57.842060 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.841514 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-home" pod "1231f78b-aaa2-4595-a11e-fcdbf984521e" (UID: "1231f78b-aaa2-4595-a11e-fcdbf984521e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:57.842060 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.841523 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-model-cache" pod "e23010a6-7ea5-40e3-b67d-f88d04c30e81" (UID: "e23010a6-7ea5-40e3-b67d-f88d04c30e81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:57.842060 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.841533 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-model-cache" pod "1231f78b-aaa2-4595-a11e-fcdbf984521e" (UID: "1231f78b-aaa2-4595-a11e-fcdbf984521e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:57.842060 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.842024 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-tmp-dir" pod "1231f78b-aaa2-4595-a11e-fcdbf984521e" (UID: "1231f78b-aaa2-4595-a11e-fcdbf984521e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:57.842603 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.842408 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-tmp-dir" pod "e23010a6-7ea5-40e3-b67d-f88d04c30e81" (UID: "e23010a6-7ea5-40e3-b67d-f88d04c30e81"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:57.842772 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.842736 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-home" pod "e23010a6-7ea5-40e3-b67d-f88d04c30e81" (UID: "e23010a6-7ea5-40e3-b67d-f88d04c30e81"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:57.845287 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.845262 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-dshm" pod "e23010a6-7ea5-40e3-b67d-f88d04c30e81" (UID: "e23010a6-7ea5-40e3-b67d-f88d04c30e81"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:57.845287 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.845274 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1231f78b-aaa2-4595-a11e-fcdbf984521e-kube-api-access-cszm2" pod "1231f78b-aaa2-4595-a11e-fcdbf984521e" (UID: "1231f78b-aaa2-4595-a11e-fcdbf984521e"). InnerVolumeSpecName "kube-api-access-cszm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:57.845628 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.845334 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-dshm" pod "1231f78b-aaa2-4595-a11e-fcdbf984521e" (UID: "1231f78b-aaa2-4595-a11e-fcdbf984521e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:57.845628 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.845538 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e23010a6-7ea5-40e3-b67d-f88d04c30e81-tls-certs" pod "e23010a6-7ea5-40e3-b67d-f88d04c30e81" (UID: "e23010a6-7ea5-40e3-b67d-f88d04c30e81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:28:57.845860 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.845828 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1231f78b-aaa2-4595-a11e-fcdbf984521e-tls-certs" pod "1231f78b-aaa2-4595-a11e-fcdbf984521e" (UID: "1231f78b-aaa2-4595-a11e-fcdbf984521e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:28:57.846971 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.846950 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23010a6-7ea5-40e3-b67d-f88d04c30e81-kube-api-access-n9g6v" pod "e23010a6-7ea5-40e3-b67d-f88d04c30e81" (UID: "e23010a6-7ea5-40e3-b67d-f88d04c30e81"). InnerVolumeSpecName "kube-api-access-n9g6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:57.942563 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.942473 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23010a6-7ea5-40e3-b67d-f88d04c30e81-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:57.942563 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.942504 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1231f78b-aaa2-4595-a11e-fcdbf984521e-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:57.942563 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.942516 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cszm2\" (UniqueName: \"kubernetes.io/projected/1231f78b-aaa2-4595-a11e-fcdbf984521e-kube-api-access-cszm2\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:57.942563 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.942528 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:57.942563 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.942539 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:57.942563 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.942549 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:57.942563 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.942560 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9g6v\" (UniqueName: \"kubernetes.io/projected/e23010a6-7ea5-40e3-b67d-f88d04c30e81-kube-api-access-n9g6v\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:57.942912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.942571 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:57.942912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.942582 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:57.942912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.942591 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:57.942912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.942601 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23010a6-7ea5-40e3-b67d-f88d04c30e81-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:57.942912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:57.942611 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1231f78b-aaa2-4595-a11e-fcdbf984521e-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:58.686785 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:58.686696 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" event={"ID":"1231f78b-aaa2-4595-a11e-fcdbf984521e","Type":"ContainerDied","Data":"e8e49855510abd06def371bf2e9be525b4875ddc4817f4d7282310b5ce1a6bc4"} Oct 29 23:28:58.686785 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:58.686755 4821 scope.go:122] "RemoveContainer" containerID="6098d9779a9f49651f7131fbe7ced8656360875db4c1c11387a3d30af7175bce" Oct 29 23:28:58.687312 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:58.686918 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7" Oct 29 23:28:58.695165 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:58.695129 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" event={"ID":"e23010a6-7ea5-40e3-b67d-f88d04c30e81","Type":"ContainerDied","Data":"3d77d759ac415f0c433e4ec29df199a4e247c72f6ba91959300d799271d4f3df"} Oct 29 23:28:58.695325 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:58.695253 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" Oct 29 23:28:58.707722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:58.707699 4821 scope.go:122] "RemoveContainer" containerID="f7593facdec14c6344cbe725bc09100f14ab15f7b00d27f159d3dfc51147e91d" Oct 29 23:28:58.719725 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:58.719697 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:28:58.745733 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:58.745703 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7"] Oct 29 23:28:58.757022 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:58.757000 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-17394a/distinct-w3-17394a-kserve-57c47678df-wtmr7"] Oct 29 23:28:58.791484 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:58.791455 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n"] Oct 29 23:28:58.795715 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:58.795692 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-17394a/simulator-w3-17394a-kserve-899fdb57f-s526n"] Oct 29 23:28:58.986995 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:58.986888 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" containerID="cri-o://0f8c172db814ec09c3af4f4e6b891049a79cac4fcb51690ab2d50a63547b4264" gracePeriod=45 Oct 29 23:28:59.019043 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.019000 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" containerID="cri-o://d3e48001cec67eafc9eb81d2d612433f1360b024ae494a4b5a306d20ca4991fe" gracePeriod=45 Oct 29 23:28:59.022490 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.022458 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" containerID="cri-o://c1b6b0bdbb4fda69b750bbdc6e3b68c5e403eb266f93c05674c6a3d86fefe5a9" gracePeriod=45 Oct 29 23:28:59.089395 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.089359 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": dial tcp 10.133.0.50:8000: connect: connection refused" Oct 29 23:28:59.097634 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.097601 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": dial tcp 10.133.0.72:8000: connect: connection refused" Oct 29 23:28:59.301820 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.301786 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 23:28:59.332602 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.332575 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 23:28:59.336036 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.336013 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 23:28:59.359011 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.358981 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-kube-api-access-n84vm\" (UniqueName: \"kubernetes.io/projected/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-kube-api-access-n84vm\") pod \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\" (UID: \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\") " Oct 29 23:28:59.359156 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.359027 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-home\" (UniqueName: \"kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-home\") pod \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\" (UID: \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\") " Oct 29 23:28:59.359156 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.359051 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-tls-certs\") pod \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\" (UID: \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\") " Oct 29 23:28:59.359299 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.359161 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-tmp-dir\") pod \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\" (UID: \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\") " Oct 29 23:28:59.359299 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.359249 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-dshm\") pod \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\" (UID: \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\") " Oct 29 23:28:59.359467 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.359451 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-model-cache\") pod \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\" (UID: \"f00ff5fc-65d2-4873-9bd5-60feb93b93d8\") " Oct 29 23:28:59.359998 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.359975 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-model-cache" pod "f00ff5fc-65d2-4873-9bd5-60feb93b93d8" (UID: "f00ff5fc-65d2-4873-9bd5-60feb93b93d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:59.360166 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.360147 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-tmp-dir" pod "f00ff5fc-65d2-4873-9bd5-60feb93b93d8" (UID: "f00ff5fc-65d2-4873-9bd5-60feb93b93d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:59.361563 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.361535 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-home" pod "f00ff5fc-65d2-4873-9bd5-60feb93b93d8" (UID: "f00ff5fc-65d2-4873-9bd5-60feb93b93d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:59.362647 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.362616 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-tls-certs" pod "f00ff5fc-65d2-4873-9bd5-60feb93b93d8" (UID: "f00ff5fc-65d2-4873-9bd5-60feb93b93d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:28:59.362858 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.362834 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-dshm" pod "f00ff5fc-65d2-4873-9bd5-60feb93b93d8" (UID: "f00ff5fc-65d2-4873-9bd5-60feb93b93d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:59.363609 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.363583 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-kube-api-access-n84vm" pod "f00ff5fc-65d2-4873-9bd5-60feb93b93d8" (UID: "f00ff5fc-65d2-4873-9bd5-60feb93b93d8"). InnerVolumeSpecName "kube-api-access-n84vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:59.461089 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.461058 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-dshm\") pod \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\" (UID: \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\") " Oct 29 23:28:59.461290 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.461130 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a152d7-e218-4f71-b9d4-265acdf8c16e-kube-api-access-npz4n\" (UniqueName: \"kubernetes.io/projected/62a152d7-e218-4f71-b9d4-265acdf8c16e-kube-api-access-npz4n\") pod \"62a152d7-e218-4f71-b9d4-265acdf8c16e\" (UID: \"62a152d7-e218-4f71-b9d4-265acdf8c16e\") " Oct 29 23:28:59.461290 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.461240 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-tls-certs\") pod \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\" (UID: \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\") " Oct 29 23:28:59.461417 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.461353 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-home\" (UniqueName: \"kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-home\") pod \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\" (UID: \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\") " Oct 29 23:28:59.461417 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.461383 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-dshm\") pod \"62a152d7-e218-4f71-b9d4-265acdf8c16e\" (UID: \"62a152d7-e218-4f71-b9d4-265acdf8c16e\") " Oct 29 23:28:59.461532 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.461426 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-tmp-dir\") pod \"62a152d7-e218-4f71-b9d4-265acdf8c16e\" (UID: \"62a152d7-e218-4f71-b9d4-265acdf8c16e\") " Oct 29 23:28:59.461532 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.461464 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-home\" (UniqueName: \"kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-home\") pod \"62a152d7-e218-4f71-b9d4-265acdf8c16e\" (UID: \"62a152d7-e218-4f71-b9d4-265acdf8c16e\") " Oct 29 23:28:59.461532 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.461503 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62a152d7-e218-4f71-b9d4-265acdf8c16e-tls-certs\" (UniqueName: \"kubernetes.io/secret/62a152d7-e218-4f71-b9d4-265acdf8c16e-tls-certs\") pod \"62a152d7-e218-4f71-b9d4-265acdf8c16e\" (UID: \"62a152d7-e218-4f71-b9d4-265acdf8c16e\") " Oct 29 23:28:59.461682 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.461595 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-model-cache\") pod \"62a152d7-e218-4f71-b9d4-265acdf8c16e\" (UID: \"62a152d7-e218-4f71-b9d4-265acdf8c16e\") " Oct 29 23:28:59.461682 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.461629 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-model-cache\") pod \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\" (UID: \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\") " Oct 29 23:28:59.461682 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.461661 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-kube-api-access-xcf6b\" (UniqueName: \"kubernetes.io/projected/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-kube-api-access-xcf6b\") pod \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\" (UID: \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\") " Oct 29 23:28:59.461837 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.461743 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-tmp-dir\") pod \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\" (UID: \"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8\") " Oct 29 23:28:59.462106 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.462077 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-model-cache" pod "62a152d7-e218-4f71-b9d4-265acdf8c16e" (UID: "62a152d7-e218-4f71-b9d4-265acdf8c16e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:59.462176 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.462138 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:59.462176 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.462157 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:59.462176 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.462173 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n84vm\" (UniqueName: \"kubernetes.io/projected/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-kube-api-access-n84vm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:59.462378 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.462186 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:59.462378 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.462218 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:59.462378 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.462232 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00ff5fc-65d2-4873-9bd5-60feb93b93d8-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:59.462580 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.462555 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-home" pod "62a152d7-e218-4f71-b9d4-265acdf8c16e" (UID: "62a152d7-e218-4f71-b9d4-265acdf8c16e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:59.462807 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.462785 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-tmp-dir" pod "62a152d7-e218-4f71-b9d4-265acdf8c16e" (UID: "62a152d7-e218-4f71-b9d4-265acdf8c16e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:59.463060 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.462940 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-tmp-dir" pod "aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" (UID: "aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:59.463060 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.462994 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-home" pod "aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" (UID: "aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:59.463060 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.463040 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-model-cache" pod "aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" (UID: "aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:59.464262 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.464108 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-dshm" pod "aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" (UID: "aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:59.471516 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.471494 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-dshm" pod "62a152d7-e218-4f71-b9d4-265acdf8c16e" (UID: "62a152d7-e218-4f71-b9d4-265acdf8c16e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:28:59.471651 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.471634 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a152d7-e218-4f71-b9d4-265acdf8c16e-kube-api-access-npz4n" pod "62a152d7-e218-4f71-b9d4-265acdf8c16e" (UID: "62a152d7-e218-4f71-b9d4-265acdf8c16e"). InnerVolumeSpecName "kube-api-access-npz4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:59.471865 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.471803 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-tls-certs" pod "aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" (UID: "aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:28:59.477350 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.477317 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62a152d7-e218-4f71-b9d4-265acdf8c16e-tls-certs" pod "62a152d7-e218-4f71-b9d4-265acdf8c16e" (UID: "62a152d7-e218-4f71-b9d4-265acdf8c16e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:28:59.477470 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.477351 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-kube-api-access-xcf6b" pod "aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" (UID: "aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8"). InnerVolumeSpecName "kube-api-access-xcf6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:28:59.563311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.563187 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:59.563311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.563259 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:59.563311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.563269 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npz4n\" (UniqueName: \"kubernetes.io/projected/62a152d7-e218-4f71-b9d4-265acdf8c16e-kube-api-access-npz4n\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:59.563311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.563278 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:59.563311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.563286 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:59.563311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.563293 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:59.563311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.563300 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:59.563311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.563307 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:59.563311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.563314 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a152d7-e218-4f71-b9d4-265acdf8c16e-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:59.563735 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.563322 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a152d7-e218-4f71-b9d4-265acdf8c16e-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:59.563735 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.563330 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:59.563735 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.563337 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcf6b\" (UniqueName: \"kubernetes.io/projected/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8-kube-api-access-xcf6b\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:28:59.704716 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.704681 4821 generic.go:356] "Generic (PLEG): container finished" podID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerID="d3e48001cec67eafc9eb81d2d612433f1360b024ae494a4b5a306d20ca4991fe" exitCode=0 Oct 29 23:28:59.705190 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.704728 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" Oct 29 23:28:59.705190 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.704764 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" event={"ID":"62a152d7-e218-4f71-b9d4-265acdf8c16e","Type":"ContainerDied","Data":"d3e48001cec67eafc9eb81d2d612433f1360b024ae494a4b5a306d20ca4991fe"} Oct 29 23:28:59.705190 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.704798 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q" event={"ID":"62a152d7-e218-4f71-b9d4-265acdf8c16e","Type":"ContainerDied","Data":"835994da901ae53002154c84551f42b95cc46835eab6ace65cc9617d64d8be98"} Oct 29 23:28:59.705190 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.704815 4821 scope.go:122] "RemoveContainer" containerID="d3e48001cec67eafc9eb81d2d612433f1360b024ae494a4b5a306d20ca4991fe" Oct 29 23:28:59.706379 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.706357 4821 generic.go:356] "Generic (PLEG): container finished" podID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerID="0f8c172db814ec09c3af4f4e6b891049a79cac4fcb51690ab2d50a63547b4264" exitCode=0 Oct 29 23:28:59.706474 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.706413 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" Oct 29 23:28:59.706531 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.706491 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" event={"ID":"f00ff5fc-65d2-4873-9bd5-60feb93b93d8","Type":"ContainerDied","Data":"0f8c172db814ec09c3af4f4e6b891049a79cac4fcb51690ab2d50a63547b4264"} Oct 29 23:28:59.706587 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.706531 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b" event={"ID":"f00ff5fc-65d2-4873-9bd5-60feb93b93d8","Type":"ContainerDied","Data":"c359dbbc89fb4feba83093e8e80f003fc12c272b840b27b7817b5081c740d3aa"} Oct 29 23:28:59.710875 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.710777 4821 generic.go:356] "Generic (PLEG): container finished" podID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerID="c1b6b0bdbb4fda69b750bbdc6e3b68c5e403eb266f93c05674c6a3d86fefe5a9" exitCode=0 Oct 29 23:28:59.710875 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.710823 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" event={"ID":"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8","Type":"ContainerDied","Data":"c1b6b0bdbb4fda69b750bbdc6e3b68c5e403eb266f93c05674c6a3d86fefe5a9"} Oct 29 23:28:59.710875 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.710850 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" event={"ID":"aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8","Type":"ContainerDied","Data":"c652939119a70307000355db063d8a9c41ecbb595b4c6ac0c41c4df3b6366f18"} Oct 29 23:28:59.710875 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.710855 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk" Oct 29 23:28:59.718950 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.718930 4821 scope.go:122] "RemoveContainer" containerID="d3e48001cec67eafc9eb81d2d612433f1360b024ae494a4b5a306d20ca4991fe" Oct 29 23:28:59.719287 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:28:59.719265 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3e48001cec67eafc9eb81d2d612433f1360b024ae494a4b5a306d20ca4991fe\": container with ID starting with d3e48001cec67eafc9eb81d2d612433f1360b024ae494a4b5a306d20ca4991fe not found: ID does not exist" containerID="d3e48001cec67eafc9eb81d2d612433f1360b024ae494a4b5a306d20ca4991fe" Oct 29 23:28:59.719415 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.719295 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e48001cec67eafc9eb81d2d612433f1360b024ae494a4b5a306d20ca4991fe"} err="failed to get container status \"d3e48001cec67eafc9eb81d2d612433f1360b024ae494a4b5a306d20ca4991fe\": rpc error: code = NotFound desc = could not find container \"d3e48001cec67eafc9eb81d2d612433f1360b024ae494a4b5a306d20ca4991fe\": container with ID starting with d3e48001cec67eafc9eb81d2d612433f1360b024ae494a4b5a306d20ca4991fe not found: ID does not exist" Oct 29 23:28:59.719415 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.719311 4821 scope.go:122] "RemoveContainer" containerID="0f8c172db814ec09c3af4f4e6b891049a79cac4fcb51690ab2d50a63547b4264" Oct 29 23:28:59.732963 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.732941 4821 scope.go:122] "RemoveContainer" containerID="0f8c172db814ec09c3af4f4e6b891049a79cac4fcb51690ab2d50a63547b4264" Oct 29 23:28:59.733249 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:28:59.733216 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f8c172db814ec09c3af4f4e6b891049a79cac4fcb51690ab2d50a63547b4264\": container with ID starting with 0f8c172db814ec09c3af4f4e6b891049a79cac4fcb51690ab2d50a63547b4264 not found: ID does not exist" containerID="0f8c172db814ec09c3af4f4e6b891049a79cac4fcb51690ab2d50a63547b4264" Oct 29 23:28:59.733313 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.733258 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f8c172db814ec09c3af4f4e6b891049a79cac4fcb51690ab2d50a63547b4264"} err="failed to get container status \"0f8c172db814ec09c3af4f4e6b891049a79cac4fcb51690ab2d50a63547b4264\": rpc error: code = NotFound desc = could not find container \"0f8c172db814ec09c3af4f4e6b891049a79cac4fcb51690ab2d50a63547b4264\": container with ID starting with 0f8c172db814ec09c3af4f4e6b891049a79cac4fcb51690ab2d50a63547b4264 not found: ID does not exist" Oct 29 23:28:59.733313 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.733277 4821 scope.go:122] "RemoveContainer" containerID="c1b6b0bdbb4fda69b750bbdc6e3b68c5e403eb266f93c05674c6a3d86fefe5a9" Oct 29 23:28:59.745403 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.745380 4821 scope.go:122] "RemoveContainer" containerID="c1b6b0bdbb4fda69b750bbdc6e3b68c5e403eb266f93c05674c6a3d86fefe5a9" Oct 29 23:28:59.745679 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:28:59.745658 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b6b0bdbb4fda69b750bbdc6e3b68c5e403eb266f93c05674c6a3d86fefe5a9\": container with ID starting with c1b6b0bdbb4fda69b750bbdc6e3b68c5e403eb266f93c05674c6a3d86fefe5a9 not found: ID does not exist" containerID="c1b6b0bdbb4fda69b750bbdc6e3b68c5e403eb266f93c05674c6a3d86fefe5a9" Oct 29 23:28:59.745777 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.745684 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b6b0bdbb4fda69b750bbdc6e3b68c5e403eb266f93c05674c6a3d86fefe5a9"} err="failed to get container status \"c1b6b0bdbb4fda69b750bbdc6e3b68c5e403eb266f93c05674c6a3d86fefe5a9\": rpc error: code = NotFound desc = could not find container \"c1b6b0bdbb4fda69b750bbdc6e3b68c5e403eb266f93c05674c6a3d86fefe5a9\": container with ID starting with c1b6b0bdbb4fda69b750bbdc6e3b68c5e403eb266f93c05674c6a3d86fefe5a9 not found: ID does not exist" Oct 29 23:28:59.763104 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.763074 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b"] Oct 29 23:28:59.768157 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.768128 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-31226c/distinct-2-w6-31226c-kserve-7ccd48c99-6qg6b"] Oct 29 23:28:59.826654 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.826574 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk"] Oct 29 23:28:59.837978 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.833627 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-31226c/simulator-w6-31226c-kserve-78ddf98fb8-2fmxk"] Oct 29 23:28:59.868253 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.868220 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q"] Oct 29 23:28:59.874054 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:28:59.874020 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-31226c/premium-w6-31226c-kserve-76b64b7dfc-qld2q"] Oct 29 23:29:00.170464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:00.170424 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" path="/var/lib/kubelet/pods/1231f78b-aaa2-4595-a11e-fcdbf984521e/volumes" Oct 29 23:29:00.170947 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:00.170923 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" path="/var/lib/kubelet/pods/62a152d7-e218-4f71-b9d4-265acdf8c16e/volumes" Oct 29 23:29:00.171454 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:00.171418 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" path="/var/lib/kubelet/pods/aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8/volumes" Oct 29 23:29:00.172169 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:00.172148 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" path="/var/lib/kubelet/pods/e23010a6-7ea5-40e3-b67d-f88d04c30e81/volumes" Oct 29 23:29:00.172619 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:00.172600 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" path="/var/lib/kubelet/pods/f00ff5fc-65d2-4873-9bd5-60feb93b93d8/volumes" Oct 29 23:29:09.917182 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.917116 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" containerID="cri-o://087d484e5ba46a82ce19d13ec113c778fca35016cb7257d93e3faccc48e99a3a" gracePeriod=45 Oct 29 23:29:09.929883 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.929827 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" containerID="cri-o://2012d9f90b11484521e675d5e5a701eb71f824e4fdece36bd82207118439ef5e" gracePeriod=45 Oct 29 23:29:09.941216 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.941172 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" containerID="cri-o://8985552fa7aa7bbb87b988b353997db125461d9a52f87deacaff33c4bfed7d6c" gracePeriod=45 Oct 29 23:29:09.953313 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.953288 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": dial tcp 10.133.0.57:8000: connect: connection refused" Oct 29 23:29:09.977110 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977074 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4471f343-bdxww"] Oct 29 23:29:09.977634 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977617 4821 cpu_manager.go:414] "containerMap: removing container" podUID="49ae9910-d400-4cec-b171-d15541fa2789" containerName="maas-api" Oct 29 23:29:09.977731 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977636 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ae9910-d400-4cec-b171-d15541fa2789" containerName="maas-api" Oct 29 23:29:09.977731 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977659 4821 cpu_manager.go:414] "containerMap: removing container" podUID="9a986cc4-88d7-432a-8cc8-f2d5371373b1" containerName="istio-proxy" Oct 29 23:29:09.977731 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977667 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a986cc4-88d7-432a-8cc8-f2d5371373b1" containerName="istio-proxy" Oct 29 23:29:09.977731 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977682 4821 cpu_manager.go:414] "containerMap: removing container" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" Oct 29 23:29:09.977731 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977690 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" Oct 29 23:29:09.977731 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977699 4821 cpu_manager.go:414] "containerMap: removing container" podUID="27c4bd01-02f4-4c07-8792-4f880626a8c0" containerName="payload-pre-processing" Oct 29 23:29:09.977731 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977706 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27c4bd01-02f4-4c07-8792-4f880626a8c0" containerName="payload-pre-processing" Oct 29 23:29:09.977731 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977720 4821 cpu_manager.go:414] "containerMap: removing container" podUID="dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" containerName="istio-proxy" Oct 29 23:29:09.977731 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977727 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977735 4821 cpu_manager.go:414] "containerMap: removing container" podUID="ce5e06a7-dc83-4262-a6a4-ead5285a1dd3" containerName="maas-api" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977743 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce5e06a7-dc83-4262-a6a4-ead5285a1dd3" containerName="maas-api" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977752 4821 cpu_manager.go:414] "containerMap: removing container" podUID="57f0a918-d7f5-4169-b2a9-106edf115d5e" containerName="main" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977761 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57f0a918-d7f5-4169-b2a9-106edf115d5e" containerName="main" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977775 4821 cpu_manager.go:414] "containerMap: removing container" podUID="7d7c0ca7-37fe-4a65-a09b-e127842adb0b" containerName="maas-api" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977782 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d7c0ca7-37fe-4a65-a09b-e127842adb0b" containerName="maas-api" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977793 4821 cpu_manager.go:414] "containerMap: removing container" podUID="cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a" containerName="main" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977799 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a" containerName="main" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977813 4821 cpu_manager.go:414] "containerMap: removing container" podUID="60b4edf6-47dc-4829-bdaf-78c062cdce78" containerName="payload-processing" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977820 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60b4edf6-47dc-4829-bdaf-78c062cdce78" containerName="payload-processing" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977831 4821 cpu_manager.go:414] "containerMap: removing container" podUID="41006044-c141-447c-8707-cf9d8d541c6b" containerName="maas-api" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977838 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41006044-c141-447c-8707-cf9d8d541c6b" containerName="maas-api" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977847 4821 cpu_manager.go:414] "containerMap: removing container" podUID="cf2ae83e-0943-4642-aa5f-2378b18bfe9d" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977854 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2ae83e-0943-4642-aa5f-2378b18bfe9d" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977864 4821 cpu_manager.go:414] "containerMap: removing container" podUID="1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0" containerName="maas-api" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977871 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0" containerName="maas-api" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977883 4821 cpu_manager.go:414] "containerMap: removing container" podUID="e438de1e-67a0-44a8-8ecd-e38ca050a541" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977890 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e438de1e-67a0-44a8-8ecd-e38ca050a541" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977899 4821 cpu_manager.go:414] "containerMap: removing container" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977906 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977917 4821 cpu_manager.go:414] "containerMap: removing container" podUID="d3c8fc4f-604c-4938-b9e1-5745ac6558c7" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977924 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3c8fc4f-604c-4938-b9e1-5745ac6558c7" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977933 4821 cpu_manager.go:414] "containerMap: removing container" podUID="c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977940 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977952 4821 cpu_manager.go:414] "containerMap: removing container" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977960 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977972 4821 cpu_manager.go:414] "containerMap: removing container" podUID="9a986cc4-88d7-432a-8cc8-f2d5371373b1" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977979 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a986cc4-88d7-432a-8cc8-f2d5371373b1" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977990 4821 cpu_manager.go:414] "containerMap: removing container" podUID="78cb6576-8c0f-4a8b-9d05-72de5333c31d" containerName="maas-api" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.977997 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78cb6576-8c0f-4a8b-9d05-72de5333c31d" containerName="maas-api" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978007 4821 cpu_manager.go:414] "containerMap: removing container" podUID="99a4a181-f0c4-438e-a5a3-f8764e83cfaf" containerName="main" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978014 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a4a181-f0c4-438e-a5a3-f8764e83cfaf" containerName="main" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978023 4821 cpu_manager.go:414] "containerMap: removing container" podUID="d560c3fe-b142-4ae7-a7da-c86260287c18" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978030 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d560c3fe-b142-4ae7-a7da-c86260287c18" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978041 4821 cpu_manager.go:414] "containerMap: removing container" podUID="df9f4a39-1043-449e-b5b3-b57c82b618b3" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978048 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df9f4a39-1043-449e-b5b3-b57c82b618b3" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978057 4821 cpu_manager.go:414] "containerMap: removing container" podUID="b74847df-b26e-4c6b-8f86-b86b5282c004" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978063 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b74847df-b26e-4c6b-8f86-b86b5282c004" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978073 4821 cpu_manager.go:414] "containerMap: removing container" podUID="d560c3fe-b142-4ae7-a7da-c86260287c18" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978080 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d560c3fe-b142-4ae7-a7da-c86260287c18" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978093 4821 cpu_manager.go:414] "containerMap: removing container" podUID="3b626717-62b8-428e-8371-9244e9f3e55f" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978101 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b626717-62b8-428e-8371-9244e9f3e55f" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978116 4821 cpu_manager.go:414] "containerMap: removing container" podUID="c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978123 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978134 4821 cpu_manager.go:414] "containerMap: removing container" podUID="073c5be5-712e-4634-82a1-9f957244d5e1" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978140 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073c5be5-712e-4634-82a1-9f957244d5e1" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978159 4821 cpu_manager.go:414] "containerMap: removing container" podUID="c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978166 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978175 4821 cpu_manager.go:414] "containerMap: removing container" podUID="e5c6effb-9b40-4c4f-aaa7-25bb59ce5469" containerName="maas-api" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978182 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c6effb-9b40-4c4f-aaa7-25bb59ce5469" containerName="maas-api" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978194 4821 cpu_manager.go:414] "containerMap: removing container" podUID="e52400e6-b2db-4dca-b23d-ad9f9a81bf44" containerName="payload-pre-processing" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978216 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52400e6-b2db-4dca-b23d-ad9f9a81bf44" containerName="payload-pre-processing" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978228 4821 cpu_manager.go:414] "containerMap: removing container" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978234 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978243 4821 cpu_manager.go:414] "containerMap: removing container" podUID="04c222af-dd88-48ed-88a0-ad27b981b410" containerName="payload-pre-processing" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978250 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04c222af-dd88-48ed-88a0-ad27b981b410" containerName="payload-pre-processing" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978265 4821 cpu_manager.go:414] "containerMap: removing container" podUID="ddb7ef9a-3948-4f03-ab50-3a9629a3c67d" containerName="payload-pre-processing" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978272 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddb7ef9a-3948-4f03-ab50-3a9629a3c67d" containerName="payload-pre-processing" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978286 4821 cpu_manager.go:414] "containerMap: removing container" podUID="da46e2ae-5388-4445-8351-bcc2204a9dcf" containerName="istio-proxy" Oct 29 23:29:09.978266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978293 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da46e2ae-5388-4445-8351-bcc2204a9dcf" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978305 4821 cpu_manager.go:414] "containerMap: removing container" podUID="c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978312 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978323 4821 cpu_manager.go:414] "containerMap: removing container" podUID="8314c4e0-da83-4173-bc1a-7fb974a7ab36" containerName="maas-api" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978330 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8314c4e0-da83-4173-bc1a-7fb974a7ab36" containerName="maas-api" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978338 4821 cpu_manager.go:414] "containerMap: removing container" podUID="631ed840-42fc-45b1-92a3-b05f747bd17f" containerName="main" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978345 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="631ed840-42fc-45b1-92a3-b05f747bd17f" containerName="main" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978359 4821 cpu_manager.go:414] "containerMap: removing container" podUID="e9c30d41-405c-49ee-931c-66f58edcd0dd" containerName="maas-api" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978366 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c30d41-405c-49ee-931c-66f58edcd0dd" containerName="maas-api" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978376 4821 cpu_manager.go:414] "containerMap: removing container" podUID="b26786c8-6181-41f2-a978-49b2123a690c" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978382 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26786c8-6181-41f2-a978-49b2123a690c" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978397 4821 cpu_manager.go:414] "containerMap: removing container" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978403 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978416 4821 cpu_manager.go:414] "containerMap: removing container" podUID="30e8d47c-27fb-40d8-86fb-eea11610630b" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978423 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30e8d47c-27fb-40d8-86fb-eea11610630b" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978450 4821 cpu_manager.go:414] "containerMap: removing container" podUID="153f58c3-7d4f-4aa1-b7a5-2f807a0b1811" containerName="payload-processing" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978458 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153f58c3-7d4f-4aa1-b7a5-2f807a0b1811" containerName="payload-processing" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978471 4821 cpu_manager.go:414] "containerMap: removing container" podUID="0fd71b7c-e947-4db2-97fb-e16431ec4a37" containerName="maas-api" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978478 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd71b7c-e947-4db2-97fb-e16431ec4a37" containerName="maas-api" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978492 4821 cpu_manager.go:414] "containerMap: removing container" podUID="8d192d0e-6dcf-4979-a503-6047643e055e" containerName="maas-api" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978499 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d192d0e-6dcf-4979-a503-6047643e055e" containerName="maas-api" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978634 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="073c5be5-712e-4634-82a1-9f957244d5e1" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978652 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978662 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c6effb-9b40-4c4f-aaa7-25bb59ce5469" containerName="maas-api" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978671 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52400e6-b2db-4dca-b23d-ad9f9a81bf44" containerName="payload-pre-processing" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978681 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="f00ff5fc-65d2-4873-9bd5-60feb93b93d8" containerName="main" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978688 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="04c222af-dd88-48ed-88a0-ad27b981b410" containerName="payload-pre-processing" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978698 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddb7ef9a-3948-4f03-ab50-3a9629a3c67d" containerName="payload-pre-processing" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978707 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="da46e2ae-5388-4445-8351-bcc2204a9dcf" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978717 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="c58b9ed9-c57e-488b-ba74-8a1b2e27ba4a" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978728 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf2ae83e-0943-4642-aa5f-2378b18bfe9d" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978735 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="8314c4e0-da83-4173-bc1a-7fb974a7ab36" containerName="maas-api" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978745 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="631ed840-42fc-45b1-92a3-b05f747bd17f" containerName="main" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978752 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c30d41-405c-49ee-931c-66f58edcd0dd" containerName="maas-api" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978761 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26786c8-6181-41f2-a978-49b2123a690c" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978770 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="1231f78b-aaa2-4595-a11e-fcdbf984521e" containerName="main" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978779 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="30e8d47c-27fb-40d8-86fb-eea11610630b" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978787 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="b74847df-b26e-4c6b-8f86-b86b5282c004" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978795 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="153f58c3-7d4f-4aa1-b7a5-2f807a0b1811" containerName="payload-processing" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978803 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd71b7c-e947-4db2-97fb-e16431ec4a37" containerName="maas-api" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978812 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d192d0e-6dcf-4979-a503-6047643e055e" containerName="maas-api" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978823 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="49ae9910-d400-4cec-b171-d15541fa2789" containerName="maas-api" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978830 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a986cc4-88d7-432a-8cc8-f2d5371373b1" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978839 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="e23010a6-7ea5-40e3-b67d-f88d04c30e81" containerName="main" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978847 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="27c4bd01-02f4-4c07-8792-4f880626a8c0" containerName="payload-pre-processing" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978856 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978865 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce5e06a7-dc83-4262-a6a4-ead5285a1dd3" containerName="maas-api" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978873 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="57f0a918-d7f5-4169-b2a9-106edf115d5e" containerName="main" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978881 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d7c0ca7-37fe-4a65-a09b-e127842adb0b" containerName="maas-api" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978891 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdb0e241-e40d-4cae-ab4c-1862ebc8dc9a" containerName="main" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978899 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="60b4edf6-47dc-4829-bdaf-78c062cdce78" containerName="payload-processing" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978906 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="41006044-c141-447c-8707-cf9d8d541c6b" containerName="maas-api" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978913 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf2ae83e-0943-4642-aa5f-2378b18bfe9d" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978921 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b2afc10-56bd-4ec9-bbbe-1ebd16f7d9f0" containerName="maas-api" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978931 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="e438de1e-67a0-44a8-8ecd-e38ca050a541" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978938 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa4898b3-8ad1-4d3a-a516-9f2a6d3468b8" containerName="main" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978946 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3c8fc4f-604c-4938-b9e1-5745ac6558c7" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978953 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978960 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a152d7-e218-4f71-b9d4-265acdf8c16e" containerName="main" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978968 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a986cc4-88d7-432a-8cc8-f2d5371373b1" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978977 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="78cb6576-8c0f-4a8b-9d05-72de5333c31d" containerName="maas-api" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978985 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a4a181-f0c4-438e-a5a3-f8764e83cfaf" containerName="main" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.978994 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="d560c3fe-b142-4ae7-a7da-c86260287c18" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.979006 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="df9f4a39-1043-449e-b5b3-b57c82b618b3" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.979014 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="d560c3fe-b142-4ae7-a7da-c86260287c18" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.979024 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b626717-62b8-428e-8371-9244e9f3e55f" containerName="istio-proxy" Oct 29 23:29:09.981061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.979032 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f5fea7-3e3e-4b5b-9fe2-8af9640dd3fe" containerName="istio-proxy" Oct 29 23:29:09.983319 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:09.983005 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4471f343-bdxww" Oct 29 23:29:10.004810 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.004783 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4471f343-bdxww"] Oct 29 23:29:10.044943 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.044904 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": dial tcp 10.133.0.65:8000: connect: connection refused" Oct 29 23:29:10.089238 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.088613 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:29:10.091416 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.091375 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:29:10.107369 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.107331 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609b76dc-c376-41a4-8eb8-43267cfa3f6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4471f343-bdxww\" (UID: \"609b76dc-c376-41a4-8eb8-43267cfa3f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4471f343-bdxww" Oct 29 23:29:10.107576 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.107405 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609b76dc-c376-41a4-8eb8-43267cfa3f6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4471f343-bdxww\" (UID: \"609b76dc-c376-41a4-8eb8-43267cfa3f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4471f343-bdxww" Oct 29 23:29:10.107576 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.107472 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vxss\" (UniqueName: \"kubernetes.io/projected/609b76dc-c376-41a4-8eb8-43267cfa3f6e-kube-api-access-7vxss\") pod \"maas-api-revoke-sub-4471f343-bdxww\" (UID: \"609b76dc-c376-41a4-8eb8-43267cfa3f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4471f343-bdxww" Oct 29 23:29:10.208568 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.208519 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609b76dc-c376-41a4-8eb8-43267cfa3f6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4471f343-bdxww\" (UID: \"609b76dc-c376-41a4-8eb8-43267cfa3f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4471f343-bdxww" Oct 29 23:29:10.208695 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.208599 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vxss\" (UniqueName: \"kubernetes.io/projected/609b76dc-c376-41a4-8eb8-43267cfa3f6e-kube-api-access-7vxss\") pod \"maas-api-revoke-sub-4471f343-bdxww\" (UID: \"609b76dc-c376-41a4-8eb8-43267cfa3f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4471f343-bdxww" Oct 29 23:29:10.208795 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.208778 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609b76dc-c376-41a4-8eb8-43267cfa3f6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4471f343-bdxww\" (UID: \"609b76dc-c376-41a4-8eb8-43267cfa3f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4471f343-bdxww" Oct 29 23:29:10.211100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.211070 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609b76dc-c376-41a4-8eb8-43267cfa3f6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4471f343-bdxww\" (UID: \"609b76dc-c376-41a4-8eb8-43267cfa3f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4471f343-bdxww" Oct 29 23:29:10.212870 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.212844 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609b76dc-c376-41a4-8eb8-43267cfa3f6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4471f343-bdxww\" (UID: \"609b76dc-c376-41a4-8eb8-43267cfa3f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4471f343-bdxww" Oct 29 23:29:10.235186 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.235158 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vxss\" (UniqueName: \"kubernetes.io/projected/609b76dc-c376-41a4-8eb8-43267cfa3f6e-kube-api-access-7vxss\") pod \"maas-api-revoke-sub-4471f343-bdxww\" (UID: \"609b76dc-c376-41a4-8eb8-43267cfa3f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4471f343-bdxww" Oct 29 23:29:10.250638 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.250617 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 23:29:10.252778 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.252754 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 23:29:10.256581 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.256564 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 23:29:10.384448 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.384409 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4471f343-bdxww" Oct 29 23:29:10.411804 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.411766 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-dshm\") pod \"9ca8086c-12ec-4fc1-b914-607bd9789748\" (UID: \"9ca8086c-12ec-4fc1-b914-607bd9789748\") " Oct 29 23:29:10.411944 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.411828 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-home\" (UniqueName: \"kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-home\") pod \"9ca8086c-12ec-4fc1-b914-607bd9789748\" (UID: \"9ca8086c-12ec-4fc1-b914-607bd9789748\") " Oct 29 23:29:10.411944 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.411875 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-model-cache\") pod \"28221151-2a2b-46b8-9fee-b658ee323a32\" (UID: \"28221151-2a2b-46b8-9fee-b658ee323a32\") " Oct 29 23:29:10.411944 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.411924 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-home\" (UniqueName: \"kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-home\") pod \"10c37b7b-4374-489e-8c71-653dd7229f60\" (UID: \"10c37b7b-4374-489e-8c71-653dd7229f60\") " Oct 29 23:29:10.412125 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.411957 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28221151-2a2b-46b8-9fee-b658ee323a32-tls-certs\" (UniqueName: \"kubernetes.io/secret/28221151-2a2b-46b8-9fee-b658ee323a32-tls-certs\") pod \"28221151-2a2b-46b8-9fee-b658ee323a32\" (UID: \"28221151-2a2b-46b8-9fee-b658ee323a32\") " Oct 29 23:29:10.412125 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.412016 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28221151-2a2b-46b8-9fee-b658ee323a32-kube-api-access-7dtwt\" (UniqueName: \"kubernetes.io/projected/28221151-2a2b-46b8-9fee-b658ee323a32-kube-api-access-7dtwt\") pod \"28221151-2a2b-46b8-9fee-b658ee323a32\" (UID: \"28221151-2a2b-46b8-9fee-b658ee323a32\") " Oct 29 23:29:10.412125 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.412064 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-model-cache\") pod \"10c37b7b-4374-489e-8c71-653dd7229f60\" (UID: \"10c37b7b-4374-489e-8c71-653dd7229f60\") " Oct 29 23:29:10.412125 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.412107 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10c37b7b-4374-489e-8c71-653dd7229f60-kube-api-access-s8n8s\" (UniqueName: \"kubernetes.io/projected/10c37b7b-4374-489e-8c71-653dd7229f60-kube-api-access-s8n8s\") pod \"10c37b7b-4374-489e-8c71-653dd7229f60\" (UID: \"10c37b7b-4374-489e-8c71-653dd7229f60\") " Oct 29 23:29:10.412426 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.412390 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-home" pod "9ca8086c-12ec-4fc1-b914-607bd9789748" (UID: "9ca8086c-12ec-4fc1-b914-607bd9789748"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:10.412772 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.412738 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-model-cache" pod "10c37b7b-4374-489e-8c71-653dd7229f60" (UID: "10c37b7b-4374-489e-8c71-653dd7229f60"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:10.412879 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.412798 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-home" pod "10c37b7b-4374-489e-8c71-653dd7229f60" (UID: "10c37b7b-4374-489e-8c71-653dd7229f60"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:10.412946 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.412895 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-tmp-dir\") pod \"10c37b7b-4374-489e-8c71-653dd7229f60\" (UID: \"10c37b7b-4374-489e-8c71-653dd7229f60\") " Oct 29 23:29:10.412946 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.412904 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-model-cache" pod "28221151-2a2b-46b8-9fee-b658ee323a32" (UID: "28221151-2a2b-46b8-9fee-b658ee323a32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:10.413069 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.412966 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-tmp-dir\") pod \"9ca8086c-12ec-4fc1-b914-607bd9789748\" (UID: \"9ca8086c-12ec-4fc1-b914-607bd9789748\") " Oct 29 23:29:10.413069 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.413036 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ca8086c-12ec-4fc1-b914-607bd9789748-tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca8086c-12ec-4fc1-b914-607bd9789748-tls-certs\") pod \"9ca8086c-12ec-4fc1-b914-607bd9789748\" (UID: \"9ca8086c-12ec-4fc1-b914-607bd9789748\") " Oct 29 23:29:10.413181 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.413066 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-model-cache\") pod \"9ca8086c-12ec-4fc1-b914-607bd9789748\" (UID: \"9ca8086c-12ec-4fc1-b914-607bd9789748\") " Oct 29 23:29:10.413181 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.413130 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-dshm\") pod \"10c37b7b-4374-489e-8c71-653dd7229f60\" (UID: \"10c37b7b-4374-489e-8c71-653dd7229f60\") " Oct 29 23:29:10.413347 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.413177 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ca8086c-12ec-4fc1-b914-607bd9789748-kube-api-access-8z4mm\" (UniqueName: \"kubernetes.io/projected/9ca8086c-12ec-4fc1-b914-607bd9789748-kube-api-access-8z4mm\") pod \"9ca8086c-12ec-4fc1-b914-607bd9789748\" (UID: \"9ca8086c-12ec-4fc1-b914-607bd9789748\") " Oct 29 23:29:10.413347 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.413311 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-tmp-dir\") pod \"28221151-2a2b-46b8-9fee-b658ee323a32\" (UID: \"28221151-2a2b-46b8-9fee-b658ee323a32\") " Oct 29 23:29:10.413464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.413356 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-home\" (UniqueName: \"kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-home\") pod \"28221151-2a2b-46b8-9fee-b658ee323a32\" (UID: \"28221151-2a2b-46b8-9fee-b658ee323a32\") " Oct 29 23:29:10.413464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.413422 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-dshm\" (UniqueName: \"kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-dshm\") pod \"28221151-2a2b-46b8-9fee-b658ee323a32\" (UID: \"28221151-2a2b-46b8-9fee-b658ee323a32\") " Oct 29 23:29:10.413464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.413426 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-tmp-dir" pod "10c37b7b-4374-489e-8c71-653dd7229f60" (UID: "10c37b7b-4374-489e-8c71-653dd7229f60"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:10.413464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.413449 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10c37b7b-4374-489e-8c71-653dd7229f60-tls-certs\" (UniqueName: \"kubernetes.io/secret/10c37b7b-4374-489e-8c71-653dd7229f60-tls-certs\") pod \"10c37b7b-4374-489e-8c71-653dd7229f60\" (UID: \"10c37b7b-4374-489e-8c71-653dd7229f60\") " Oct 29 23:29:10.415865 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.413972 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:10.415865 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.413995 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:10.415865 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.414009 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:10.415865 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.414021 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:10.415865 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.414033 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:10.415865 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.414539 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-home" pod "28221151-2a2b-46b8-9fee-b658ee323a32" (UID: "28221151-2a2b-46b8-9fee-b658ee323a32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:10.415865 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.414714 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-tmp-dir" pod "28221151-2a2b-46b8-9fee-b658ee323a32" (UID: "28221151-2a2b-46b8-9fee-b658ee323a32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:10.415865 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.415469 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28221151-2a2b-46b8-9fee-b658ee323a32-kube-api-access-7dtwt" pod "28221151-2a2b-46b8-9fee-b658ee323a32" (UID: "28221151-2a2b-46b8-9fee-b658ee323a32"). InnerVolumeSpecName "kube-api-access-7dtwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:10.415865 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.415679 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-model-cache" pod "9ca8086c-12ec-4fc1-b914-607bd9789748" (UID: "9ca8086c-12ec-4fc1-b914-607bd9789748"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:10.419621 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.419593 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-dshm" pod "9ca8086c-12ec-4fc1-b914-607bd9789748" (UID: "9ca8086c-12ec-4fc1-b914-607bd9789748"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:10.419787 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.419746 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10c37b7b-4374-489e-8c71-653dd7229f60-tls-certs" pod "10c37b7b-4374-489e-8c71-653dd7229f60" (UID: "10c37b7b-4374-489e-8c71-653dd7229f60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:10.419863 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.419795 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-tmp-dir" pod "9ca8086c-12ec-4fc1-b914-607bd9789748" (UID: "9ca8086c-12ec-4fc1-b914-607bd9789748"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:10.420154 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.420125 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28221151-2a2b-46b8-9fee-b658ee323a32-tls-certs" pod "28221151-2a2b-46b8-9fee-b658ee323a32" (UID: "28221151-2a2b-46b8-9fee-b658ee323a32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:10.420267 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.420233 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca8086c-12ec-4fc1-b914-607bd9789748-kube-api-access-8z4mm" pod "9ca8086c-12ec-4fc1-b914-607bd9789748" (UID: "9ca8086c-12ec-4fc1-b914-607bd9789748"). InnerVolumeSpecName "kube-api-access-8z4mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:10.420336 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.420313 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c37b7b-4374-489e-8c71-653dd7229f60-kube-api-access-s8n8s" pod "10c37b7b-4374-489e-8c71-653dd7229f60" (UID: "10c37b7b-4374-489e-8c71-653dd7229f60"). InnerVolumeSpecName "kube-api-access-s8n8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:10.420414 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.420371 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-dshm" pod "28221151-2a2b-46b8-9fee-b658ee323a32" (UID: "28221151-2a2b-46b8-9fee-b658ee323a32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:10.420662 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.420631 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-dshm" pod "10c37b7b-4374-489e-8c71-653dd7229f60" (UID: "10c37b7b-4374-489e-8c71-653dd7229f60"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:10.420731 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.420696 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ca8086c-12ec-4fc1-b914-607bd9789748-tls-certs" pod "9ca8086c-12ec-4fc1-b914-607bd9789748" (UID: "9ca8086c-12ec-4fc1-b914-607bd9789748"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:10.530331 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.520582 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:10.530331 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.520628 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28221151-2a2b-46b8-9fee-b658ee323a32-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:10.530331 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.520643 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dtwt\" (UniqueName: \"kubernetes.io/projected/28221151-2a2b-46b8-9fee-b658ee323a32-kube-api-access-7dtwt\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:10.530331 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.520658 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8n8s\" (UniqueName: \"kubernetes.io/projected/10c37b7b-4374-489e-8c71-653dd7229f60-kube-api-access-s8n8s\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:10.530331 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.520670 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:10.530331 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.520684 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca8086c-12ec-4fc1-b914-607bd9789748-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:10.530331 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.520696 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca8086c-12ec-4fc1-b914-607bd9789748-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:10.530331 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.520708 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c37b7b-4374-489e-8c71-653dd7229f60-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:10.530331 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.520719 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z4mm\" (UniqueName: \"kubernetes.io/projected/9ca8086c-12ec-4fc1-b914-607bd9789748-kube-api-access-8z4mm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:10.530331 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.520730 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:10.530331 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.520741 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:10.530331 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.520751 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28221151-2a2b-46b8-9fee-b658ee323a32-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:10.530331 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.520762 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c37b7b-4374-489e-8c71-653dd7229f60-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:10.699944 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.699915 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4471f343-bdxww"] Oct 29 23:29:10.701424 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:29:10.701395 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod609b76dc_c376_41a4_8eb8_43267cfa3f6e.slice/crio-858e656240a15474161c1488b0c7a2698f54d107687b22774748d9418502eda7 WatchSource:0}: Error finding container 858e656240a15474161c1488b0c7a2698f54d107687b22774748d9418502eda7: Status 404 returned error can't find the container with id 858e656240a15474161c1488b0c7a2698f54d107687b22774748d9418502eda7 Oct 29 23:29:10.825189 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.825156 4821 generic.go:356] "Generic (PLEG): container finished" podID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerID="2012d9f90b11484521e675d5e5a701eb71f824e4fdece36bd82207118439ef5e" exitCode=0 Oct 29 23:29:10.825391 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.825308 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" Oct 29 23:29:10.825391 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.825322 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" event={"ID":"9ca8086c-12ec-4fc1-b914-607bd9789748","Type":"ContainerDied","Data":"2012d9f90b11484521e675d5e5a701eb71f824e4fdece36bd82207118439ef5e"} Oct 29 23:29:10.825391 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.825376 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr" event={"ID":"9ca8086c-12ec-4fc1-b914-607bd9789748","Type":"ContainerDied","Data":"6dc36a2d48a91f85adb9cca96a15295cea3626ae30cb9cfee19df073b72e5ce4"} Oct 29 23:29:10.825560 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.825402 4821 scope.go:122] "RemoveContainer" containerID="2012d9f90b11484521e675d5e5a701eb71f824e4fdece36bd82207118439ef5e" Oct 29 23:29:10.826790 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.826762 4821 generic.go:356] "Generic (PLEG): container finished" podID="10c37b7b-4374-489e-8c71-653dd7229f60" containerID="087d484e5ba46a82ce19d13ec113c778fca35016cb7257d93e3faccc48e99a3a" exitCode=0 Oct 29 23:29:10.826932 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.826829 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" Oct 29 23:29:10.826932 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.826853 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" event={"ID":"10c37b7b-4374-489e-8c71-653dd7229f60","Type":"ContainerDied","Data":"087d484e5ba46a82ce19d13ec113c778fca35016cb7257d93e3faccc48e99a3a"} Oct 29 23:29:10.826932 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.826885 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf" event={"ID":"10c37b7b-4374-489e-8c71-653dd7229f60","Type":"ContainerDied","Data":"c6a721646d17f0edb24563d1bac3ce82f46fdc00c309b0b78db78aee44f749d4"} Oct 29 23:29:10.828315 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.828290 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4471f343-bdxww" event={"ID":"609b76dc-c376-41a4-8eb8-43267cfa3f6e","Type":"ContainerStarted","Data":"858e656240a15474161c1488b0c7a2698f54d107687b22774748d9418502eda7"} Oct 29 23:29:10.829606 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.829576 4821 generic.go:356] "Generic (PLEG): container finished" podID="28221151-2a2b-46b8-9fee-b658ee323a32" containerID="8985552fa7aa7bbb87b988b353997db125461d9a52f87deacaff33c4bfed7d6c" exitCode=0 Oct 29 23:29:10.829606 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.829603 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" event={"ID":"28221151-2a2b-46b8-9fee-b658ee323a32","Type":"ContainerDied","Data":"8985552fa7aa7bbb87b988b353997db125461d9a52f87deacaff33c4bfed7d6c"} Oct 29 23:29:10.829792 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.829637 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" event={"ID":"28221151-2a2b-46b8-9fee-b658ee323a32","Type":"ContainerDied","Data":"02ba8db9510b89fd73a0bb83714a7913a8eec19072cbd0c9bcabb5d1d9deb1ef"} Oct 29 23:29:10.829792 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.829641 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4" Oct 29 23:29:10.838146 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.838120 4821 scope.go:122] "RemoveContainer" containerID="2012d9f90b11484521e675d5e5a701eb71f824e4fdece36bd82207118439ef5e" Oct 29 23:29:10.838456 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:29:10.838420 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2012d9f90b11484521e675d5e5a701eb71f824e4fdece36bd82207118439ef5e\": container with ID starting with 2012d9f90b11484521e675d5e5a701eb71f824e4fdece36bd82207118439ef5e not found: ID does not exist" containerID="2012d9f90b11484521e675d5e5a701eb71f824e4fdece36bd82207118439ef5e" Oct 29 23:29:10.838507 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.838450 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2012d9f90b11484521e675d5e5a701eb71f824e4fdece36bd82207118439ef5e"} err="failed to get container status \"2012d9f90b11484521e675d5e5a701eb71f824e4fdece36bd82207118439ef5e\": rpc error: code = NotFound desc = could not find container \"2012d9f90b11484521e675d5e5a701eb71f824e4fdece36bd82207118439ef5e\": container with ID starting with 2012d9f90b11484521e675d5e5a701eb71f824e4fdece36bd82207118439ef5e not found: ID does not exist" Oct 29 23:29:10.838507 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.838467 4821 scope.go:122] "RemoveContainer" containerID="087d484e5ba46a82ce19d13ec113c778fca35016cb7257d93e3faccc48e99a3a" Oct 29 23:29:10.848321 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.848302 4821 scope.go:122] "RemoveContainer" containerID="087d484e5ba46a82ce19d13ec113c778fca35016cb7257d93e3faccc48e99a3a" Oct 29 23:29:10.848595 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:29:10.848577 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"087d484e5ba46a82ce19d13ec113c778fca35016cb7257d93e3faccc48e99a3a\": container with ID starting with 087d484e5ba46a82ce19d13ec113c778fca35016cb7257d93e3faccc48e99a3a not found: ID does not exist" containerID="087d484e5ba46a82ce19d13ec113c778fca35016cb7257d93e3faccc48e99a3a" Oct 29 23:29:10.848647 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.848603 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087d484e5ba46a82ce19d13ec113c778fca35016cb7257d93e3faccc48e99a3a"} err="failed to get container status \"087d484e5ba46a82ce19d13ec113c778fca35016cb7257d93e3faccc48e99a3a\": rpc error: code = NotFound desc = could not find container \"087d484e5ba46a82ce19d13ec113c778fca35016cb7257d93e3faccc48e99a3a\": container with ID starting with 087d484e5ba46a82ce19d13ec113c778fca35016cb7257d93e3faccc48e99a3a not found: ID does not exist" Oct 29 23:29:10.848647 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.848619 4821 scope.go:122] "RemoveContainer" containerID="8985552fa7aa7bbb87b988b353997db125461d9a52f87deacaff33c4bfed7d6c" Oct 29 23:29:10.857808 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.857784 4821 scope.go:122] "RemoveContainer" containerID="8985552fa7aa7bbb87b988b353997db125461d9a52f87deacaff33c4bfed7d6c" Oct 29 23:29:10.858045 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:29:10.858027 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8985552fa7aa7bbb87b988b353997db125461d9a52f87deacaff33c4bfed7d6c\": container with ID starting with 8985552fa7aa7bbb87b988b353997db125461d9a52f87deacaff33c4bfed7d6c not found: ID does not exist" containerID="8985552fa7aa7bbb87b988b353997db125461d9a52f87deacaff33c4bfed7d6c" Oct 29 23:29:10.858097 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.858051 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8985552fa7aa7bbb87b988b353997db125461d9a52f87deacaff33c4bfed7d6c"} err="failed to get container status \"8985552fa7aa7bbb87b988b353997db125461d9a52f87deacaff33c4bfed7d6c\": rpc error: code = NotFound desc = could not find container \"8985552fa7aa7bbb87b988b353997db125461d9a52f87deacaff33c4bfed7d6c\": container with ID starting with 8985552fa7aa7bbb87b988b353997db125461d9a52f87deacaff33c4bfed7d6c not found: ID does not exist" Oct 29 23:29:10.880735 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.880709 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4"] Oct 29 23:29:10.894169 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.894145 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-b2921b/simulator-w5-b2921b-kserve-5555547b64-xs9h4"] Oct 29 23:29:10.929726 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.929689 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr"] Oct 29 23:29:10.936885 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.936864 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-b2921b/distinct-w5-b2921b-kserve-5d966f7484-2c4lr"] Oct 29 23:29:10.975164 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.975082 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf"] Oct 29 23:29:10.979165 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:10.979141 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-b2921b/distinct-2-w5-b2921b-kserve-5cbffbc847-kz4cf"] Oct 29 23:29:11.842170 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:11.842074 4821 generic.go:356] "Generic (PLEG): container finished" podID="609b76dc-c376-41a4-8eb8-43267cfa3f6e" containerID="273f90d2b522635a198eae39d16c8f83d0c0c5fa41cb27f012ccbb78358c1ec3" exitCode=0 Oct 29 23:29:11.842170 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:11.842117 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4471f343-bdxww" event={"ID":"609b76dc-c376-41a4-8eb8-43267cfa3f6e","Type":"ContainerDied","Data":"273f90d2b522635a198eae39d16c8f83d0c0c5fa41cb27f012ccbb78358c1ec3"} Oct 29 23:29:12.169479 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:12.169447 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" path="/var/lib/kubelet/pods/10c37b7b-4374-489e-8c71-653dd7229f60/volumes" Oct 29 23:29:12.169872 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:12.169862 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" path="/var/lib/kubelet/pods/28221151-2a2b-46b8-9fee-b658ee323a32/volumes" Oct 29 23:29:12.170243 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:12.170230 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" path="/var/lib/kubelet/pods/9ca8086c-12ec-4fc1-b914-607bd9789748/volumes" Oct 29 23:29:12.988278 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:12.988248 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4471f343-bdxww" Oct 29 23:29:13.152567 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:13.152525 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/609b76dc-c376-41a4-8eb8-43267cfa3f6e-kube-api-access-7vxss\" (UniqueName: \"kubernetes.io/projected/609b76dc-c376-41a4-8eb8-43267cfa3f6e-kube-api-access-7vxss\") pod \"609b76dc-c376-41a4-8eb8-43267cfa3f6e\" (UID: \"609b76dc-c376-41a4-8eb8-43267cfa3f6e\") " Oct 29 23:29:13.152730 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:13.152594 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/609b76dc-c376-41a4-8eb8-43267cfa3f6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609b76dc-c376-41a4-8eb8-43267cfa3f6e-maas-api-xks-ca\") pod \"609b76dc-c376-41a4-8eb8-43267cfa3f6e\" (UID: \"609b76dc-c376-41a4-8eb8-43267cfa3f6e\") " Oct 29 23:29:13.152730 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:13.152664 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/609b76dc-c376-41a4-8eb8-43267cfa3f6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609b76dc-c376-41a4-8eb8-43267cfa3f6e-maas-api-openshift-ca\") pod \"609b76dc-c376-41a4-8eb8-43267cfa3f6e\" (UID: \"609b76dc-c376-41a4-8eb8-43267cfa3f6e\") " Oct 29 23:29:13.152974 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:13.152937 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/609b76dc-c376-41a4-8eb8-43267cfa3f6e-maas-api-openshift-ca" pod "609b76dc-c376-41a4-8eb8-43267cfa3f6e" (UID: "609b76dc-c376-41a4-8eb8-43267cfa3f6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:13.153110 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:13.153098 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609b76dc-c376-41a4-8eb8-43267cfa3f6e-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:13.154815 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:13.154793 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/609b76dc-c376-41a4-8eb8-43267cfa3f6e-maas-api-xks-ca" pod "609b76dc-c376-41a4-8eb8-43267cfa3f6e" (UID: "609b76dc-c376-41a4-8eb8-43267cfa3f6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:13.154815 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:13.154804 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609b76dc-c376-41a4-8eb8-43267cfa3f6e-kube-api-access-7vxss" pod "609b76dc-c376-41a4-8eb8-43267cfa3f6e" (UID: "609b76dc-c376-41a4-8eb8-43267cfa3f6e"). InnerVolumeSpecName "kube-api-access-7vxss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:13.253875 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:13.253839 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vxss\" (UniqueName: \"kubernetes.io/projected/609b76dc-c376-41a4-8eb8-43267cfa3f6e-kube-api-access-7vxss\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:13.253875 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:13.253865 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609b76dc-c376-41a4-8eb8-43267cfa3f6e-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:13.858533 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:13.858508 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4471f343-bdxww" Oct 29 23:29:13.858711 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:13.858537 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4471f343-bdxww" event={"ID":"609b76dc-c376-41a4-8eb8-43267cfa3f6e","Type":"ContainerDied","Data":"858e656240a15474161c1488b0c7a2698f54d107687b22774748d9418502eda7"} Oct 29 23:29:13.858711 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:13.858585 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="858e656240a15474161c1488b0c7a2698f54d107687b22774748d9418502eda7" Oct 29 23:29:21.630273 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.630232 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4decf31-v7bg5"] Oct 29 23:29:21.630966 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.630941 4821 cpu_manager.go:414] "containerMap: removing container" podUID="b74847df-b26e-4c6b-8f86-b86b5282c004" containerName="istio-proxy" Oct 29 23:29:21.630966 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.630963 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b74847df-b26e-4c6b-8f86-b86b5282c004" containerName="istio-proxy" Oct 29 23:29:21.631227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.630978 4821 cpu_manager.go:414] "containerMap: removing container" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" Oct 29 23:29:21.631227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.630986 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" Oct 29 23:29:21.631227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.631032 4821 cpu_manager.go:414] "containerMap: removing container" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" Oct 29 23:29:21.631227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.631040 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" Oct 29 23:29:21.631227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.631052 4821 cpu_manager.go:414] "containerMap: removing container" podUID="dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" containerName="istio-proxy" Oct 29 23:29:21.631227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.631057 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" containerName="istio-proxy" Oct 29 23:29:21.631227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.631065 4821 cpu_manager.go:414] "containerMap: removing container" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" Oct 29 23:29:21.631227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.631070 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" Oct 29 23:29:21.631227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.631077 4821 cpu_manager.go:414] "containerMap: removing container" podUID="609b76dc-c376-41a4-8eb8-43267cfa3f6e" containerName="revoke-keys" Oct 29 23:29:21.631227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.631082 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="609b76dc-c376-41a4-8eb8-43267cfa3f6e" containerName="revoke-keys" Oct 29 23:29:21.631227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.631088 4821 cpu_manager.go:414] "containerMap: removing container" podUID="cf2ae83e-0943-4642-aa5f-2378b18bfe9d" containerName="istio-proxy" Oct 29 23:29:21.631227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.631093 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2ae83e-0943-4642-aa5f-2378b18bfe9d" containerName="istio-proxy" Oct 29 23:29:21.631227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.631175 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="b74847df-b26e-4c6b-8f86-b86b5282c004" containerName="istio-proxy" Oct 29 23:29:21.631227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.631183 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="10c37b7b-4374-489e-8c71-653dd7229f60" containerName="main" Oct 29 23:29:21.631227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.631192 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc080a2a-5786-4e10-adbc-9b47e3ea4e1b" containerName="istio-proxy" Oct 29 23:29:21.631227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.631214 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="609b76dc-c376-41a4-8eb8-43267cfa3f6e" containerName="revoke-keys" Oct 29 23:29:21.631227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.631219 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="28221151-2a2b-46b8-9fee-b658ee323a32" containerName="main" Oct 29 23:29:21.631227 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.631227 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca8086c-12ec-4fc1-b914-607bd9789748" containerName="main" Oct 29 23:29:21.634674 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.634656 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4decf31-v7bg5" Oct 29 23:29:21.655365 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.655343 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4decf31-v7bg5"] Oct 29 23:29:21.655708 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.655684 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m599\" (UniqueName: \"kubernetes.io/projected/cf7a517b-fc28-4e18-ab4c-afcb95d27a96-kube-api-access-8m599\") pod \"maas-api-revoke-sub-d4decf31-v7bg5\" (UID: \"cf7a517b-fc28-4e18-ab4c-afcb95d27a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4decf31-v7bg5" Oct 29 23:29:21.655860 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.655840 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7a517b-fc28-4e18-ab4c-afcb95d27a96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4decf31-v7bg5\" (UID: \"cf7a517b-fc28-4e18-ab4c-afcb95d27a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4decf31-v7bg5" Oct 29 23:29:21.656023 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.656004 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7a517b-fc28-4e18-ab4c-afcb95d27a96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4decf31-v7bg5\" (UID: \"cf7a517b-fc28-4e18-ab4c-afcb95d27a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4decf31-v7bg5" Oct 29 23:29:21.666997 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.665421 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:29:21.666997 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.665615 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:29:21.757382 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.757350 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7a517b-fc28-4e18-ab4c-afcb95d27a96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4decf31-v7bg5\" (UID: \"cf7a517b-fc28-4e18-ab4c-afcb95d27a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4decf31-v7bg5" Oct 29 23:29:21.757603 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.757427 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7a517b-fc28-4e18-ab4c-afcb95d27a96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4decf31-v7bg5\" (UID: \"cf7a517b-fc28-4e18-ab4c-afcb95d27a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4decf31-v7bg5" Oct 29 23:29:21.757603 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.757463 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m599\" (UniqueName: \"kubernetes.io/projected/cf7a517b-fc28-4e18-ab4c-afcb95d27a96-kube-api-access-8m599\") pod \"maas-api-revoke-sub-d4decf31-v7bg5\" (UID: \"cf7a517b-fc28-4e18-ab4c-afcb95d27a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4decf31-v7bg5" Oct 29 23:29:21.758116 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.758097 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7a517b-fc28-4e18-ab4c-afcb95d27a96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4decf31-v7bg5\" (UID: \"cf7a517b-fc28-4e18-ab4c-afcb95d27a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4decf31-v7bg5" Oct 29 23:29:21.759900 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.759881 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7a517b-fc28-4e18-ab4c-afcb95d27a96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4decf31-v7bg5\" (UID: \"cf7a517b-fc28-4e18-ab4c-afcb95d27a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4decf31-v7bg5" Oct 29 23:29:21.782381 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.782356 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m599\" (UniqueName: \"kubernetes.io/projected/cf7a517b-fc28-4e18-ab4c-afcb95d27a96-kube-api-access-8m599\") pod \"maas-api-revoke-sub-d4decf31-v7bg5\" (UID: \"cf7a517b-fc28-4e18-ab4c-afcb95d27a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4decf31-v7bg5" Oct 29 23:29:21.948161 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:21.948085 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4decf31-v7bg5" Oct 29 23:29:22.257128 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:22.257102 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4decf31-v7bg5"] Oct 29 23:29:22.259109 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:29:22.259078 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf7a517b_fc28_4e18_ab4c_afcb95d27a96.slice/crio-162a5980497b57740b32ecefa72fcf5ec4d4d22bec58e365c7050f65dfa07d91 WatchSource:0}: Error finding container 162a5980497b57740b32ecefa72fcf5ec4d4d22bec58e365c7050f65dfa07d91: Status 404 returned error can't find the container with id 162a5980497b57740b32ecefa72fcf5ec4d4d22bec58e365c7050f65dfa07d91 Oct 29 23:29:22.944879 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:22.944843 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4decf31-v7bg5" event={"ID":"cf7a517b-fc28-4e18-ab4c-afcb95d27a96","Type":"ContainerStarted","Data":"a2f77234aa5350642813cb686e80d1459429437897071bb71b20b6e97da190cb"} Oct 29 23:29:22.945311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:22.944909 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4decf31-v7bg5" event={"ID":"cf7a517b-fc28-4e18-ab4c-afcb95d27a96","Type":"ContainerStarted","Data":"162a5980497b57740b32ecefa72fcf5ec4d4d22bec58e365c7050f65dfa07d91"} Oct 29 23:29:22.989035 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:22.988972 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4decf31-v7bg5" podStartSLOduration=1.98895511 podStartE2EDuration="1.98895511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:22.987671298 +0000 UTC m=+2816.928219207" watchObservedRunningTime="2026-10-29 23:29:22.98895511 +0000 UTC m=+2816.929503023" Oct 29 23:29:23.887485 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:29:23.885085 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:29:23.953765 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:23.953729 4821 generic.go:356] "Generic (PLEG): container finished" podID="cf7a517b-fc28-4e18-ab4c-afcb95d27a96" containerID="a2f77234aa5350642813cb686e80d1459429437897071bb71b20b6e97da190cb" exitCode=0 Oct 29 23:29:23.954161 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:23.953765 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4decf31-v7bg5" event={"ID":"cf7a517b-fc28-4e18-ab4c-afcb95d27a96","Type":"ContainerDied","Data":"a2f77234aa5350642813cb686e80d1459429437897071bb71b20b6e97da190cb"} Oct 29 23:29:25.109108 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:25.109077 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4decf31-v7bg5" Oct 29 23:29:25.236696 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:25.236616 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf7a517b-fc28-4e18-ab4c-afcb95d27a96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7a517b-fc28-4e18-ab4c-afcb95d27a96-maas-api-openshift-ca\") pod \"cf7a517b-fc28-4e18-ab4c-afcb95d27a96\" (UID: \"cf7a517b-fc28-4e18-ab4c-afcb95d27a96\") " Oct 29 23:29:25.236876 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:25.236721 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf7a517b-fc28-4e18-ab4c-afcb95d27a96-kube-api-access-8m599\" (UniqueName: \"kubernetes.io/projected/cf7a517b-fc28-4e18-ab4c-afcb95d27a96-kube-api-access-8m599\") pod \"cf7a517b-fc28-4e18-ab4c-afcb95d27a96\" (UID: \"cf7a517b-fc28-4e18-ab4c-afcb95d27a96\") " Oct 29 23:29:25.236876 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:25.236833 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf7a517b-fc28-4e18-ab4c-afcb95d27a96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7a517b-fc28-4e18-ab4c-afcb95d27a96-maas-api-xks-ca\") pod \"cf7a517b-fc28-4e18-ab4c-afcb95d27a96\" (UID: \"cf7a517b-fc28-4e18-ab4c-afcb95d27a96\") " Oct 29 23:29:25.236999 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:25.236943 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf7a517b-fc28-4e18-ab4c-afcb95d27a96-maas-api-openshift-ca" pod "cf7a517b-fc28-4e18-ab4c-afcb95d27a96" (UID: "cf7a517b-fc28-4e18-ab4c-afcb95d27a96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:25.237142 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:25.237123 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7a517b-fc28-4e18-ab4c-afcb95d27a96-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:25.238985 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:25.238961 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf7a517b-fc28-4e18-ab4c-afcb95d27a96-maas-api-xks-ca" pod "cf7a517b-fc28-4e18-ab4c-afcb95d27a96" (UID: "cf7a517b-fc28-4e18-ab4c-afcb95d27a96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:29:25.239100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:25.238990 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf7a517b-fc28-4e18-ab4c-afcb95d27a96-kube-api-access-8m599" pod "cf7a517b-fc28-4e18-ab4c-afcb95d27a96" (UID: "cf7a517b-fc28-4e18-ab4c-afcb95d27a96"). InnerVolumeSpecName "kube-api-access-8m599". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:25.338873 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:25.338836 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7a517b-fc28-4e18-ab4c-afcb95d27a96-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:25.338873 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:25.338872 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m599\" (UniqueName: \"kubernetes.io/projected/cf7a517b-fc28-4e18-ab4c-afcb95d27a96-kube-api-access-8m599\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:25.967910 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:25.967874 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4decf31-v7bg5" Oct 29 23:29:25.968122 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:25.967903 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4decf31-v7bg5" event={"ID":"cf7a517b-fc28-4e18-ab4c-afcb95d27a96","Type":"ContainerDied","Data":"162a5980497b57740b32ecefa72fcf5ec4d4d22bec58e365c7050f65dfa07d91"} Oct 29 23:29:25.968122 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:25.967960 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="162a5980497b57740b32ecefa72fcf5ec4d4d22bec58e365c7050f65dfa07d91" Oct 29 23:29:32.160977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:32.160941 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb"] Oct 29 23:29:32.161394 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:32.161144 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" podUID="fc867e1c-3627-44c9-bc0a-94765ae3d2d7" containerName="istio-proxy" containerID="cri-o://4ec8a37dd43adcd287b69538870ca8c96325fe5f7bfe7f135e0b221d6b91bbbf" gracePeriod=30 Oct 29 23:29:37.435855 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.435790 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 23:29:37.521313 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.521234 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-workload-socket\") pod \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " Oct 29 23:29:37.521313 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.521304 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-data\") pod \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " Oct 29 23:29:37.521559 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.521350 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istiod-ca-cert\") pod \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " Oct 29 23:29:37.521559 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.521388 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-token\" (UniqueName: \"kubernetes.io/projected/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-token\") pod \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " Oct 29 23:29:37.521559 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.521418 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-credential-socket\") pod \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " Oct 29 23:29:37.521559 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.521446 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-podinfo\") pod \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " Oct 29 23:29:37.521559 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.521499 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-envoy\") pod \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " Oct 29 23:29:37.521843 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.521593 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-workload-socket" pod "fc867e1c-3627-44c9-bc0a-94765ae3d2d7" (UID: "fc867e1c-3627-44c9-bc0a-94765ae3d2d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:37.521843 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.521604 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-kube-api-access-c2qgm\" (UniqueName: \"kubernetes.io/projected/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-kube-api-access-c2qgm\") pod \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " Oct 29 23:29:37.521843 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.521635 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-data" pod "fc867e1c-3627-44c9-bc0a-94765ae3d2d7" (UID: "fc867e1c-3627-44c9-bc0a-94765ae3d2d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:37.521843 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.521743 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-workload-certs\") pod \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\" (UID: \"fc867e1c-3627-44c9-bc0a-94765ae3d2d7\") " Oct 29 23:29:37.521843 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.521813 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istiod-ca-cert" pod "fc867e1c-3627-44c9-bc0a-94765ae3d2d7" (UID: "fc867e1c-3627-44c9-bc0a-94765ae3d2d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:37.522120 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.522044 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-workload-certs" pod "fc867e1c-3627-44c9-bc0a-94765ae3d2d7" (UID: "fc867e1c-3627-44c9-bc0a-94765ae3d2d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:37.522120 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.522058 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-credential-socket" pod "fc867e1c-3627-44c9-bc0a-94765ae3d2d7" (UID: "fc867e1c-3627-44c9-bc0a-94765ae3d2d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:37.522443 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.522424 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:37.522703 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.522688 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:37.522910 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.522886 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:37.523042 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.522911 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:37.523042 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.522927 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:37.524247 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.524193 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-envoy" pod "fc867e1c-3627-44c9-bc0a-94765ae3d2d7" (UID: "fc867e1c-3627-44c9-bc0a-94765ae3d2d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:37.524445 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.524416 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-podinfo" pod "fc867e1c-3627-44c9-bc0a-94765ae3d2d7" (UID: "fc867e1c-3627-44c9-bc0a-94765ae3d2d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:29:37.524551 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.524463 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-kube-api-access-c2qgm" pod "fc867e1c-3627-44c9-bc0a-94765ae3d2d7" (UID: "fc867e1c-3627-44c9-bc0a-94765ae3d2d7"). InnerVolumeSpecName "kube-api-access-c2qgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:37.524551 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.524496 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-token" pod "fc867e1c-3627-44c9-bc0a-94765ae3d2d7" (UID: "fc867e1c-3627-44c9-bc0a-94765ae3d2d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:37.623973 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.623941 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2qgm\" (UniqueName: \"kubernetes.io/projected/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-kube-api-access-c2qgm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:37.623973 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.623971 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:37.624172 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.623984 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:37.624172 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:37.623995 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc867e1c-3627-44c9-bc0a-94765ae3d2d7-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:38.050989 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:38.050955 4821 generic.go:356] "Generic (PLEG): container finished" podID="fc867e1c-3627-44c9-bc0a-94765ae3d2d7" containerID="4ec8a37dd43adcd287b69538870ca8c96325fe5f7bfe7f135e0b221d6b91bbbf" exitCode=0 Oct 29 23:29:38.051166 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:38.051012 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" Oct 29 23:29:38.051166 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:38.051082 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" event={"ID":"fc867e1c-3627-44c9-bc0a-94765ae3d2d7","Type":"ContainerDied","Data":"4ec8a37dd43adcd287b69538870ca8c96325fe5f7bfe7f135e0b221d6b91bbbf"} Oct 29 23:29:38.051166 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:38.051128 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb" event={"ID":"fc867e1c-3627-44c9-bc0a-94765ae3d2d7","Type":"ContainerDied","Data":"6697834f3994647c2c438c153da666d81cd7b59f27811a0e555e65a0068eb6cb"} Oct 29 23:29:38.051166 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:38.051150 4821 scope.go:122] "RemoveContainer" containerID="4ec8a37dd43adcd287b69538870ca8c96325fe5f7bfe7f135e0b221d6b91bbbf" Oct 29 23:29:38.063570 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:38.063551 4821 scope.go:122] "RemoveContainer" containerID="4ec8a37dd43adcd287b69538870ca8c96325fe5f7bfe7f135e0b221d6b91bbbf" Oct 29 23:29:38.063873 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:29:38.063853 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec8a37dd43adcd287b69538870ca8c96325fe5f7bfe7f135e0b221d6b91bbbf\": container with ID starting with 4ec8a37dd43adcd287b69538870ca8c96325fe5f7bfe7f135e0b221d6b91bbbf not found: ID does not exist" containerID="4ec8a37dd43adcd287b69538870ca8c96325fe5f7bfe7f135e0b221d6b91bbbf" Oct 29 23:29:38.063917 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:38.063883 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec8a37dd43adcd287b69538870ca8c96325fe5f7bfe7f135e0b221d6b91bbbf"} err="failed to get container status \"4ec8a37dd43adcd287b69538870ca8c96325fe5f7bfe7f135e0b221d6b91bbbf\": rpc error: code = NotFound desc = could not find container \"4ec8a37dd43adcd287b69538870ca8c96325fe5f7bfe7f135e0b221d6b91bbbf\": container with ID starting with 4ec8a37dd43adcd287b69538870ca8c96325fe5f7bfe7f135e0b221d6b91bbbf not found: ID does not exist" Oct 29 23:29:38.121890 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:38.121855 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb"] Oct 29 23:29:38.134809 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:38.134783 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-17394a-openshift-default-6db4bfdb5c-qbfkb"] Oct 29 23:29:38.169118 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:38.169093 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc867e1c-3627-44c9-bc0a-94765ae3d2d7" path="/var/lib/kubelet/pods/fc867e1c-3627-44c9-bc0a-94765ae3d2d7/volumes" Oct 29 23:29:53.550848 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:53.550812 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774"] Oct 29 23:29:53.551532 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:53.551497 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" podUID="ae79f378-d055-4e08-bd3a-1e75fcc38882" containerName="istio-proxy" containerID="cri-o://f13f2b0b590d9d8bc228994b0979fe93cd11c43b58855606a64cb24fe97f5f48" gracePeriod=30 Oct 29 23:29:53.885359 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:29:53.885314 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:29:58.826543 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:58.826475 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 23:29:58.979574 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:58.979539 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-podinfo\") pod \"ae79f378-d055-4e08-bd3a-1e75fcc38882\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " Oct 29 23:29:58.979748 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:58.979589 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae79f378-d055-4e08-bd3a-1e75fcc38882-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae79f378-d055-4e08-bd3a-1e75fcc38882-istiod-ca-cert\") pod \"ae79f378-d055-4e08-bd3a-1e75fcc38882\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " Oct 29 23:29:58.979748 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:58.979661 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-token\" (UniqueName: \"kubernetes.io/projected/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-token\") pod \"ae79f378-d055-4e08-bd3a-1e75fcc38882\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " Oct 29 23:29:58.979748 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:58.979696 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-envoy\") pod \"ae79f378-d055-4e08-bd3a-1e75fcc38882\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " Oct 29 23:29:58.979859 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:58.979808 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-credential-socket\") pod \"ae79f378-d055-4e08-bd3a-1e75fcc38882\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " Oct 29 23:29:58.979859 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:58.979852 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae79f378-d055-4e08-bd3a-1e75fcc38882-kube-api-access-t7zkz\" (UniqueName: \"kubernetes.io/projected/ae79f378-d055-4e08-bd3a-1e75fcc38882-kube-api-access-t7zkz\") pod \"ae79f378-d055-4e08-bd3a-1e75fcc38882\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " Oct 29 23:29:58.979972 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:58.979897 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-workload-certs\") pod \"ae79f378-d055-4e08-bd3a-1e75fcc38882\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " Oct 29 23:29:58.979972 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:58.979940 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-workload-socket\") pod \"ae79f378-d055-4e08-bd3a-1e75fcc38882\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " Oct 29 23:29:58.979972 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:58.979968 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-data\") pod \"ae79f378-d055-4e08-bd3a-1e75fcc38882\" (UID: \"ae79f378-d055-4e08-bd3a-1e75fcc38882\") " Oct 29 23:29:58.980148 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:58.980072 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-credential-socket" pod "ae79f378-d055-4e08-bd3a-1e75fcc38882" (UID: "ae79f378-d055-4e08-bd3a-1e75fcc38882"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:58.980243 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:58.980169 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-workload-certs" pod "ae79f378-d055-4e08-bd3a-1e75fcc38882" (UID: "ae79f378-d055-4e08-bd3a-1e75fcc38882"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:58.980243 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:58.980088 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae79f378-d055-4e08-bd3a-1e75fcc38882-istiod-ca-cert" pod "ae79f378-d055-4e08-bd3a-1e75fcc38882" (UID: "ae79f378-d055-4e08-bd3a-1e75fcc38882"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:29:58.980403 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:58.980380 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-workload-socket" pod "ae79f378-d055-4e08-bd3a-1e75fcc38882" (UID: "ae79f378-d055-4e08-bd3a-1e75fcc38882"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:58.980772 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:58.980747 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-data" pod "ae79f378-d055-4e08-bd3a-1e75fcc38882" (UID: "ae79f378-d055-4e08-bd3a-1e75fcc38882"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:58.980948 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:58.980761 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:58.981092 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:58.980961 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:58.981092 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:58.980980 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae79f378-d055-4e08-bd3a-1e75fcc38882-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:58.981092 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:58.980993 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:58.982252 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:58.982188 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-podinfo" pod "ae79f378-d055-4e08-bd3a-1e75fcc38882" (UID: "ae79f378-d055-4e08-bd3a-1e75fcc38882"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:29:58.982345 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:58.982241 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-envoy" pod "ae79f378-d055-4e08-bd3a-1e75fcc38882" (UID: "ae79f378-d055-4e08-bd3a-1e75fcc38882"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:29:58.982345 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:58.982334 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-token" pod "ae79f378-d055-4e08-bd3a-1e75fcc38882" (UID: "ae79f378-d055-4e08-bd3a-1e75fcc38882"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:58.982737 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:58.982714 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae79f378-d055-4e08-bd3a-1e75fcc38882-kube-api-access-t7zkz" pod "ae79f378-d055-4e08-bd3a-1e75fcc38882" (UID: "ae79f378-d055-4e08-bd3a-1e75fcc38882"). InnerVolumeSpecName "kube-api-access-t7zkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:29:59.082180 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:59.082101 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7zkz\" (UniqueName: \"kubernetes.io/projected/ae79f378-d055-4e08-bd3a-1e75fcc38882-kube-api-access-t7zkz\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:59.082180 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:59.082131 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:59.082180 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:59.082140 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:59.082180 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:59.082147 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:59.082180 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:59.082155 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae79f378-d055-4e08-bd3a-1e75fcc38882-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:29:59.208628 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:59.208594 4821 generic.go:356] "Generic (PLEG): container finished" podID="ae79f378-d055-4e08-bd3a-1e75fcc38882" containerID="f13f2b0b590d9d8bc228994b0979fe93cd11c43b58855606a64cb24fe97f5f48" exitCode=0 Oct 29 23:29:59.208821 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:59.208672 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" Oct 29 23:29:59.208821 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:59.208680 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" event={"ID":"ae79f378-d055-4e08-bd3a-1e75fcc38882","Type":"ContainerDied","Data":"f13f2b0b590d9d8bc228994b0979fe93cd11c43b58855606a64cb24fe97f5f48"} Oct 29 23:29:59.208821 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:59.208732 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774" event={"ID":"ae79f378-d055-4e08-bd3a-1e75fcc38882","Type":"ContainerDied","Data":"bb96fcb876c295bf73155ecf548fc0d6a4391570a24bfda35d9b31c8dced51db"} Oct 29 23:29:59.208821 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:59.208754 4821 scope.go:122] "RemoveContainer" containerID="f13f2b0b590d9d8bc228994b0979fe93cd11c43b58855606a64cb24fe97f5f48" Oct 29 23:29:59.220662 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:59.220636 4821 scope.go:122] "RemoveContainer" containerID="f13f2b0b590d9d8bc228994b0979fe93cd11c43b58855606a64cb24fe97f5f48" Oct 29 23:29:59.220939 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:29:59.220921 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f13f2b0b590d9d8bc228994b0979fe93cd11c43b58855606a64cb24fe97f5f48\": container with ID starting with f13f2b0b590d9d8bc228994b0979fe93cd11c43b58855606a64cb24fe97f5f48 not found: ID does not exist" containerID="f13f2b0b590d9d8bc228994b0979fe93cd11c43b58855606a64cb24fe97f5f48" Oct 29 23:29:59.220988 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:59.220948 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13f2b0b590d9d8bc228994b0979fe93cd11c43b58855606a64cb24fe97f5f48"} err="failed to get container status \"f13f2b0b590d9d8bc228994b0979fe93cd11c43b58855606a64cb24fe97f5f48\": rpc error: code = NotFound desc = could not find container \"f13f2b0b590d9d8bc228994b0979fe93cd11c43b58855606a64cb24fe97f5f48\": container with ID starting with f13f2b0b590d9d8bc228994b0979fe93cd11c43b58855606a64cb24fe97f5f48 not found: ID does not exist" Oct 29 23:29:59.265044 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:59.265012 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774"] Oct 29 23:29:59.268751 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:29:59.268722 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b2921b-openshift-default-5547c88fff-9p774"] Oct 29 23:30:00.149024 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.148987 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l"] Oct 29 23:30:00.149601 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.149494 4821 cpu_manager.go:414] "containerMap: removing container" podUID="cf7a517b-fc28-4e18-ab4c-afcb95d27a96" containerName="revoke-keys" Oct 29 23:30:00.149601 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.149508 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf7a517b-fc28-4e18-ab4c-afcb95d27a96" containerName="revoke-keys" Oct 29 23:30:00.149601 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.149523 4821 cpu_manager.go:414] "containerMap: removing container" podUID="fc867e1c-3627-44c9-bc0a-94765ae3d2d7" containerName="istio-proxy" Oct 29 23:30:00.149601 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.149529 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc867e1c-3627-44c9-bc0a-94765ae3d2d7" containerName="istio-proxy" Oct 29 23:30:00.149601 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.149541 4821 cpu_manager.go:414] "containerMap: removing container" podUID="ae79f378-d055-4e08-bd3a-1e75fcc38882" containerName="istio-proxy" Oct 29 23:30:00.149601 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.149546 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae79f378-d055-4e08-bd3a-1e75fcc38882" containerName="istio-proxy" Oct 29 23:30:00.149799 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.149619 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf7a517b-fc28-4e18-ab4c-afcb95d27a96" containerName="revoke-keys" Oct 29 23:30:00.149799 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.149628 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc867e1c-3627-44c9-bc0a-94765ae3d2d7" containerName="istio-proxy" Oct 29 23:30:00.149799 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.149634 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae79f378-d055-4e08-bd3a-1e75fcc38882" containerName="istio-proxy" Oct 29 23:30:00.154155 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.154138 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l" Oct 29 23:30:00.171456 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.171397 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae79f378-d055-4e08-bd3a-1e75fcc38882" path="/var/lib/kubelet/pods/ae79f378-d055-4e08-bd3a-1e75fcc38882/volumes" Oct 29 23:30:00.173154 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.173124 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt"] Oct 29 23:30:00.177689 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.177657 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt" Oct 29 23:30:00.188989 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.188962 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l"] Oct 29 23:30:00.197446 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.197424 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf"] Oct 29 23:30:00.201426 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.201402 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf" Oct 29 23:30:00.218085 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.218052 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt"] Oct 29 23:30:00.222604 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.222575 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s"] Oct 29 23:30:00.228924 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.228894 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s" Oct 29 23:30:00.231280 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.231227 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf"] Oct 29 23:30:00.236154 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.236124 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s"] Oct 29 23:30:00.268218 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.268175 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:30:00.296290 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.296262 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkd8d\" (UniqueName: \"kubernetes.io/projected/ca5af571-204a-4e17-890f-3a3d929c0827-kube-api-access-lkd8d\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l\" (UID: \"ca5af571-204a-4e17-890f-3a3d929c0827\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l" Oct 29 23:30:00.296290 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.296293 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmjlh\" (UniqueName: \"kubernetes.io/projected/7ed236b2-7d1a-46a6-bbb9-55683420c3ed-kube-api-access-wmjlh\") pod \"maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt\" (UID: \"7ed236b2-7d1a-46a6-bbb9-55683420c3ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt" Oct 29 23:30:00.296516 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.296316 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mrv5\" (UniqueName: \"kubernetes.io/projected/99fcc999-240e-4254-9926-d1af30f9f8f8-kube-api-access-5mrv5\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf\" (UID: \"99fcc999-240e-4254-9926-d1af30f9f8f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf" Oct 29 23:30:00.296516 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.296477 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5af571-204a-4e17-890f-3a3d929c0827-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l\" (UID: \"ca5af571-204a-4e17-890f-3a3d929c0827\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l" Oct 29 23:30:00.296612 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.296529 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed236b2-7d1a-46a6-bbb9-55683420c3ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt\" (UID: \"7ed236b2-7d1a-46a6-bbb9-55683420c3ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt" Oct 29 23:30:00.296652 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.296623 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99fcc999-240e-4254-9926-d1af30f9f8f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf\" (UID: \"99fcc999-240e-4254-9926-d1af30f9f8f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf" Oct 29 23:30:00.397973 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.397937 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p57m9\" (UniqueName: \"kubernetes.io/projected/bc852c3e-5277-4f30-9063-f4901fdca67e-kube-api-access-p57m9\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s\" (UID: \"bc852c3e-5277-4f30-9063-f4901fdca67e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s" Oct 29 23:30:00.398183 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.397988 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5af571-204a-4e17-890f-3a3d929c0827-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l\" (UID: \"ca5af571-204a-4e17-890f-3a3d929c0827\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l" Oct 29 23:30:00.398183 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.398096 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed236b2-7d1a-46a6-bbb9-55683420c3ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt\" (UID: \"7ed236b2-7d1a-46a6-bbb9-55683420c3ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt" Oct 29 23:30:00.398183 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.398140 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99fcc999-240e-4254-9926-d1af30f9f8f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf\" (UID: \"99fcc999-240e-4254-9926-d1af30f9f8f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf" Oct 29 23:30:00.398367 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.398229 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc852c3e-5277-4f30-9063-f4901fdca67e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s\" (UID: \"bc852c3e-5277-4f30-9063-f4901fdca67e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s" Oct 29 23:30:00.398367 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.398295 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkd8d\" (UniqueName: \"kubernetes.io/projected/ca5af571-204a-4e17-890f-3a3d929c0827-kube-api-access-lkd8d\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l\" (UID: \"ca5af571-204a-4e17-890f-3a3d929c0827\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l" Oct 29 23:30:00.398367 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.398320 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmjlh\" (UniqueName: \"kubernetes.io/projected/7ed236b2-7d1a-46a6-bbb9-55683420c3ed-kube-api-access-wmjlh\") pod \"maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt\" (UID: \"7ed236b2-7d1a-46a6-bbb9-55683420c3ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt" Oct 29 23:30:00.398367 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.398350 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mrv5\" (UniqueName: \"kubernetes.io/projected/99fcc999-240e-4254-9926-d1af30f9f8f8-kube-api-access-5mrv5\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf\" (UID: \"99fcc999-240e-4254-9926-d1af30f9f8f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf" Oct 29 23:30:00.398750 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.398730 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5af571-204a-4e17-890f-3a3d929c0827-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l\" (UID: \"ca5af571-204a-4e17-890f-3a3d929c0827\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l" Oct 29 23:30:00.398912 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.398896 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99fcc999-240e-4254-9926-d1af30f9f8f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf\" (UID: \"99fcc999-240e-4254-9926-d1af30f9f8f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf" Oct 29 23:30:00.398967 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.398905 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed236b2-7d1a-46a6-bbb9-55683420c3ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt\" (UID: \"7ed236b2-7d1a-46a6-bbb9-55683420c3ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt" Oct 29 23:30:00.423174 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.423102 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkd8d\" (UniqueName: \"kubernetes.io/projected/ca5af571-204a-4e17-890f-3a3d929c0827-kube-api-access-lkd8d\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l\" (UID: \"ca5af571-204a-4e17-890f-3a3d929c0827\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l" Oct 29 23:30:00.423409 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.423378 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmjlh\" (UniqueName: \"kubernetes.io/projected/7ed236b2-7d1a-46a6-bbb9-55683420c3ed-kube-api-access-wmjlh\") pod \"maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt\" (UID: \"7ed236b2-7d1a-46a6-bbb9-55683420c3ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt" Oct 29 23:30:00.423554 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.423531 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mrv5\" (UniqueName: \"kubernetes.io/projected/99fcc999-240e-4254-9926-d1af30f9f8f8-kube-api-access-5mrv5\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf\" (UID: \"99fcc999-240e-4254-9926-d1af30f9f8f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf" Oct 29 23:30:00.467100 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.467071 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l" Oct 29 23:30:00.499478 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.499431 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc852c3e-5277-4f30-9063-f4901fdca67e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s\" (UID: \"bc852c3e-5277-4f30-9063-f4901fdca67e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s" Oct 29 23:30:00.499640 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.499591 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p57m9\" (UniqueName: \"kubernetes.io/projected/bc852c3e-5277-4f30-9063-f4901fdca67e-kube-api-access-p57m9\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s\" (UID: \"bc852c3e-5277-4f30-9063-f4901fdca67e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s" Oct 29 23:30:00.500115 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.500085 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc852c3e-5277-4f30-9063-f4901fdca67e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s\" (UID: \"bc852c3e-5277-4f30-9063-f4901fdca67e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s" Oct 29 23:30:00.501059 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.501037 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt" Oct 29 23:30:00.522921 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.522892 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf" Oct 29 23:30:00.525169 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.525111 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p57m9\" (UniqueName: \"kubernetes.io/projected/bc852c3e-5277-4f30-9063-f4901fdca67e-kube-api-access-p57m9\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s\" (UID: \"bc852c3e-5277-4f30-9063-f4901fdca67e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s" Oct 29 23:30:00.546891 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.546855 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s" Oct 29 23:30:00.776314 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.776288 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l"] Oct 29 23:30:00.777969 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:30:00.777941 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5af571_204a_4e17_890f_3a3d929c0827.slice/crio-299ed0ce014232bdc563c844fab940f57651b7ff9b26a4cc175a34385c23d628 WatchSource:0}: Error finding container 299ed0ce014232bdc563c844fab940f57651b7ff9b26a4cc175a34385c23d628: Status 404 returned error can't find the container with id 299ed0ce014232bdc563c844fab940f57651b7ff9b26a4cc175a34385c23d628 Oct 29 23:30:00.830465 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.830428 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt"] Oct 29 23:30:00.834687 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:30:00.834661 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed236b2_7d1a_46a6_bbb9_55683420c3ed.slice/crio-3d3335fadea9a3a5d09ea7ce8ab643e0170e8028512d125141053f9859d66a5e WatchSource:0}: Error finding container 3d3335fadea9a3a5d09ea7ce8ab643e0170e8028512d125141053f9859d66a5e: Status 404 returned error can't find the container with id 3d3335fadea9a3a5d09ea7ce8ab643e0170e8028512d125141053f9859d66a5e Oct 29 23:30:00.895250 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.895044 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf"] Oct 29 23:30:00.899329 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:00.899303 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s"] Oct 29 23:30:00.903135 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:30:00.903100 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc852c3e_5277_4f30_9063_f4901fdca67e.slice/crio-6a0014d0f4b1d88044362859512b0aa43652bfd2377935f79765b69b92aa7eb9 WatchSource:0}: Error finding container 6a0014d0f4b1d88044362859512b0aa43652bfd2377935f79765b69b92aa7eb9: Status 404 returned error can't find the container with id 6a0014d0f4b1d88044362859512b0aa43652bfd2377935f79765b69b92aa7eb9 Oct 29 23:30:01.235657 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:01.235136 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l" event={"ID":"ca5af571-204a-4e17-890f-3a3d929c0827","Type":"ContainerStarted","Data":"ece211c0508f2d74055b92d579ef50745bf869443ae9978f37c2a64584bedeb8"} Oct 29 23:30:01.235657 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:01.235219 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l" event={"ID":"ca5af571-204a-4e17-890f-3a3d929c0827","Type":"ContainerStarted","Data":"299ed0ce014232bdc563c844fab940f57651b7ff9b26a4cc175a34385c23d628"} Oct 29 23:30:01.237104 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:01.237078 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf" event={"ID":"99fcc999-240e-4254-9926-d1af30f9f8f8","Type":"ContainerStarted","Data":"c894425ce1e0b23ba437123cdfc6aa780ee6f661f1d472c347090f7cdebc8dc9"} Oct 29 23:30:01.238590 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:01.238563 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s" event={"ID":"bc852c3e-5277-4f30-9063-f4901fdca67e","Type":"ContainerStarted","Data":"6a0014d0f4b1d88044362859512b0aa43652bfd2377935f79765b69b92aa7eb9"} Oct 29 23:30:01.240317 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:01.240288 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt" event={"ID":"7ed236b2-7d1a-46a6-bbb9-55683420c3ed","Type":"ContainerStarted","Data":"3d3335fadea9a3a5d09ea7ce8ab643e0170e8028512d125141053f9859d66a5e"} Oct 29 23:30:01.283295 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:01.283240 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l" podStartSLOduration=1.283225165 podStartE2EDuration="1.283225165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:01.281189636 +0000 UTC m=+2855.221737544" watchObservedRunningTime="2026-10-29 23:30:01.283225165 +0000 UTC m=+2855.223773071" Oct 29 23:30:01.324639 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:01.324481 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt" podStartSLOduration=1.324462984 podStartE2EDuration="1.324462984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:01.323087902 +0000 UTC m=+2855.263635809" watchObservedRunningTime="2026-10-29 23:30:01.324462984 +0000 UTC m=+2855.265010896" Oct 29 23:30:02.250451 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:02.250362 4821 generic.go:356] "Generic (PLEG): container finished" podID="7ed236b2-7d1a-46a6-bbb9-55683420c3ed" containerID="2f145d6dbdfa453fb97e8f7594e4e4b4f31718bf1a02b57c9b2521569d0b115f" exitCode=60 Oct 29 23:30:02.250867 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:02.250442 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt" event={"ID":"7ed236b2-7d1a-46a6-bbb9-55683420c3ed","Type":"ContainerDied","Data":"2f145d6dbdfa453fb97e8f7594e4e4b4f31718bf1a02b57c9b2521569d0b115f"} Oct 29 23:30:02.250867 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:02.250701 4821 scope.go:122] "RemoveContainer" containerID="2f145d6dbdfa453fb97e8f7594e4e4b4f31718bf1a02b57c9b2521569d0b115f" Oct 29 23:30:02.252817 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:02.252794 4821 generic.go:356] "Generic (PLEG): container finished" podID="ca5af571-204a-4e17-890f-3a3d929c0827" containerID="ece211c0508f2d74055b92d579ef50745bf869443ae9978f37c2a64584bedeb8" exitCode=60 Oct 29 23:30:02.252939 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:02.252893 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l" event={"ID":"ca5af571-204a-4e17-890f-3a3d929c0827","Type":"ContainerDied","Data":"ece211c0508f2d74055b92d579ef50745bf869443ae9978f37c2a64584bedeb8"} Oct 29 23:30:02.253141 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:02.253126 4821 scope.go:122] "RemoveContainer" containerID="ece211c0508f2d74055b92d579ef50745bf869443ae9978f37c2a64584bedeb8" Oct 29 23:30:02.255537 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:02.255511 4821 generic.go:356] "Generic (PLEG): container finished" podID="99fcc999-240e-4254-9926-d1af30f9f8f8" containerID="e94996597a0dfd691a545971c733b1ea6f6e64bf80df42433cba5ba81d2f226e" exitCode=60 Oct 29 23:30:02.255835 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:02.255800 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf" event={"ID":"99fcc999-240e-4254-9926-d1af30f9f8f8","Type":"ContainerDied","Data":"e94996597a0dfd691a545971c733b1ea6f6e64bf80df42433cba5ba81d2f226e"} Oct 29 23:30:02.255957 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:02.255917 4821 scope.go:122] "RemoveContainer" containerID="e94996597a0dfd691a545971c733b1ea6f6e64bf80df42433cba5ba81d2f226e" Oct 29 23:30:02.257753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:02.257730 4821 generic.go:356] "Generic (PLEG): container finished" podID="bc852c3e-5277-4f30-9063-f4901fdca67e" containerID="df95221917c30e07272f033b9b6b643e925be1aca26378836f606539d9e61dc8" exitCode=60 Oct 29 23:30:02.257868 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:02.257804 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s" event={"ID":"bc852c3e-5277-4f30-9063-f4901fdca67e","Type":"ContainerDied","Data":"df95221917c30e07272f033b9b6b643e925be1aca26378836f606539d9e61dc8"} Oct 29 23:30:02.258019 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:02.258003 4821 scope.go:122] "RemoveContainer" containerID="df95221917c30e07272f033b9b6b643e925be1aca26378836f606539d9e61dc8" Oct 29 23:30:03.266785 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:03.266751 4821 generic.go:356] "Generic (PLEG): container finished" podID="7ed236b2-7d1a-46a6-bbb9-55683420c3ed" containerID="c46b6c1a4fad3a051132ab83b9d44109ab25e98da2450cbe672a2a304fe0f445" exitCode=60 Oct 29 23:30:03.267308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:03.266789 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt" event={"ID":"7ed236b2-7d1a-46a6-bbb9-55683420c3ed","Type":"ContainerDied","Data":"c46b6c1a4fad3a051132ab83b9d44109ab25e98da2450cbe672a2a304fe0f445"} Oct 29 23:30:03.267308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:03.266849 4821 scope.go:122] "RemoveContainer" containerID="2f145d6dbdfa453fb97e8f7594e4e4b4f31718bf1a02b57c9b2521569d0b115f" Oct 29 23:30:03.267308 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:03.267137 4821 scope.go:122] "RemoveContainer" containerID="c46b6c1a4fad3a051132ab83b9d44109ab25e98da2450cbe672a2a304fe0f445" Oct 29 23:30:03.267533 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:30:03.267492 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt_odh-ai-gateway-infra(7ed236b2-7d1a-46a6-bbb9-55683420c3ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt" podUID="7ed236b2-7d1a-46a6-bbb9-55683420c3ed" Oct 29 23:30:03.269123 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:03.269100 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l" event={"ID":"ca5af571-204a-4e17-890f-3a3d929c0827","Type":"ContainerStarted","Data":"34ea57bfec29f84976727255cb6ffde2fc0bc4b544e498bb4df999a0a13d29c6"} Oct 29 23:30:03.271066 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:03.271042 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf" event={"ID":"99fcc999-240e-4254-9926-d1af30f9f8f8","Type":"ContainerStarted","Data":"65b707d566a7676188bc60e5c68cced9db6b2e4d5fd1acb1d9692187971ed2c0"} Oct 29 23:30:03.273030 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:03.272974 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s" event={"ID":"bc852c3e-5277-4f30-9063-f4901fdca67e","Type":"ContainerStarted","Data":"54bd3c27bb143cd2d722c5d525eeb1e42234c2f2f8129377cf2766bd454151e5"} Oct 29 23:30:03.354905 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:03.354855 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf" podStartSLOduration=3.354842386 podStartE2EDuration="3.354842386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:03.353290678 +0000 UTC m=+2857.293838585" watchObservedRunningTime="2026-10-29 23:30:03.354842386 +0000 UTC m=+2857.295390295" Oct 29 23:30:03.437668 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:03.437616 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s" podStartSLOduration=3.437600794 podStartE2EDuration="3.437600794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:03.436855396 +0000 UTC m=+2857.377403305" watchObservedRunningTime="2026-10-29 23:30:03.437600794 +0000 UTC m=+2857.378148738" Oct 29 23:30:04.281250 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:04.281191 4821 generic.go:356] "Generic (PLEG): container finished" podID="bc852c3e-5277-4f30-9063-f4901fdca67e" containerID="54bd3c27bb143cd2d722c5d525eeb1e42234c2f2f8129377cf2766bd454151e5" exitCode=60 Oct 29 23:30:04.281761 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:04.281252 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s" event={"ID":"bc852c3e-5277-4f30-9063-f4901fdca67e","Type":"ContainerDied","Data":"54bd3c27bb143cd2d722c5d525eeb1e42234c2f2f8129377cf2766bd454151e5"} Oct 29 23:30:04.281761 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:04.281372 4821 scope.go:122] "RemoveContainer" containerID="df95221917c30e07272f033b9b6b643e925be1aca26378836f606539d9e61dc8" Oct 29 23:30:04.281761 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:04.281677 4821 scope.go:122] "RemoveContainer" containerID="54bd3c27bb143cd2d722c5d525eeb1e42234c2f2f8129377cf2766bd454151e5" Oct 29 23:30:04.282042 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:30:04.282016 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s_odh-ai-gateway-infra(bc852c3e-5277-4f30-9063-f4901fdca67e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s" podUID="bc852c3e-5277-4f30-9063-f4901fdca67e" Oct 29 23:30:04.283630 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:04.283606 4821 scope.go:122] "RemoveContainer" containerID="c46b6c1a4fad3a051132ab83b9d44109ab25e98da2450cbe672a2a304fe0f445" Oct 29 23:30:04.283908 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:30:04.283889 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt_odh-ai-gateway-infra(7ed236b2-7d1a-46a6-bbb9-55683420c3ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt" podUID="7ed236b2-7d1a-46a6-bbb9-55683420c3ed" Oct 29 23:30:04.285213 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:04.285173 4821 generic.go:356] "Generic (PLEG): container finished" podID="ca5af571-204a-4e17-890f-3a3d929c0827" containerID="34ea57bfec29f84976727255cb6ffde2fc0bc4b544e498bb4df999a0a13d29c6" exitCode=60 Oct 29 23:30:04.285526 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:04.285492 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l" event={"ID":"ca5af571-204a-4e17-890f-3a3d929c0827","Type":"ContainerDied","Data":"34ea57bfec29f84976727255cb6ffde2fc0bc4b544e498bb4df999a0a13d29c6"} Oct 29 23:30:04.285695 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:04.285588 4821 scope.go:122] "RemoveContainer" containerID="34ea57bfec29f84976727255cb6ffde2fc0bc4b544e498bb4df999a0a13d29c6" Oct 29 23:30:04.285807 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:30:04.285786 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l_odh-ai-gateway-infra(ca5af571-204a-4e17-890f-3a3d929c0827)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l" podUID="ca5af571-204a-4e17-890f-3a3d929c0827" Oct 29 23:30:04.287379 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:04.287359 4821 generic.go:356] "Generic (PLEG): container finished" podID="99fcc999-240e-4254-9926-d1af30f9f8f8" containerID="65b707d566a7676188bc60e5c68cced9db6b2e4d5fd1acb1d9692187971ed2c0" exitCode=60 Oct 29 23:30:04.287485 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:04.287423 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf" event={"ID":"99fcc999-240e-4254-9926-d1af30f9f8f8","Type":"ContainerDied","Data":"65b707d566a7676188bc60e5c68cced9db6b2e4d5fd1acb1d9692187971ed2c0"} Oct 29 23:30:04.287639 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:04.287627 4821 scope.go:122] "RemoveContainer" containerID="65b707d566a7676188bc60e5c68cced9db6b2e4d5fd1acb1d9692187971ed2c0" Oct 29 23:30:04.287811 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:30:04.287796 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf_odh-ai-gateway-infra(99fcc999-240e-4254-9926-d1af30f9f8f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf" podUID="99fcc999-240e-4254-9926-d1af30f9f8f8" Oct 29 23:30:04.294385 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:04.294369 4821 scope.go:122] "RemoveContainer" containerID="ece211c0508f2d74055b92d579ef50745bf869443ae9978f37c2a64584bedeb8" Oct 29 23:30:04.307257 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:04.307240 4821 scope.go:122] "RemoveContainer" containerID="e94996597a0dfd691a545971c733b1ea6f6e64bf80df42433cba5ba81d2f226e" Oct 29 23:30:05.297105 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:05.297080 4821 scope.go:122] "RemoveContainer" containerID="54bd3c27bb143cd2d722c5d525eeb1e42234c2f2f8129377cf2766bd454151e5" Oct 29 23:30:05.297508 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:30:05.297355 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s_odh-ai-gateway-infra(bc852c3e-5277-4f30-9063-f4901fdca67e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s" podUID="bc852c3e-5277-4f30-9063-f4901fdca67e" Oct 29 23:30:05.298803 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:05.298781 4821 scope.go:122] "RemoveContainer" containerID="34ea57bfec29f84976727255cb6ffde2fc0bc4b544e498bb4df999a0a13d29c6" Oct 29 23:30:05.299011 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:30:05.298987 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l_odh-ai-gateway-infra(ca5af571-204a-4e17-890f-3a3d929c0827)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l" podUID="ca5af571-204a-4e17-890f-3a3d929c0827" Oct 29 23:30:05.300380 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:05.300364 4821 scope.go:122] "RemoveContainer" containerID="65b707d566a7676188bc60e5c68cced9db6b2e4d5fd1acb1d9692187971ed2c0" Oct 29 23:30:05.300564 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:30:05.300536 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf_odh-ai-gateway-infra(99fcc999-240e-4254-9926-d1af30f9f8f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf" podUID="99fcc999-240e-4254-9926-d1af30f9f8f8" Oct 29 23:30:15.162765 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:15.162728 4821 scope.go:122] "RemoveContainer" containerID="c46b6c1a4fad3a051132ab83b9d44109ab25e98da2450cbe672a2a304fe0f445" Oct 29 23:30:16.166062 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:16.166036 4821 scope.go:122] "RemoveContainer" containerID="54bd3c27bb143cd2d722c5d525eeb1e42234c2f2f8129377cf2766bd454151e5" Oct 29 23:30:16.381901 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:16.381865 4821 generic.go:356] "Generic (PLEG): container finished" podID="7ed236b2-7d1a-46a6-bbb9-55683420c3ed" containerID="366c910ef659824023df1eacce8b4838d2829e2dcec4ed988a360937e009b2a2" exitCode=60 Oct 29 23:30:16.382055 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:16.381934 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt" event={"ID":"7ed236b2-7d1a-46a6-bbb9-55683420c3ed","Type":"ContainerDied","Data":"366c910ef659824023df1eacce8b4838d2829e2dcec4ed988a360937e009b2a2"} Oct 29 23:30:16.382055 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:16.382001 4821 scope.go:122] "RemoveContainer" containerID="c46b6c1a4fad3a051132ab83b9d44109ab25e98da2450cbe672a2a304fe0f445" Oct 29 23:30:16.382422 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:16.382401 4821 scope.go:122] "RemoveContainer" containerID="366c910ef659824023df1eacce8b4838d2829e2dcec4ed988a360937e009b2a2" Oct 29 23:30:16.382738 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:30:16.382714 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt_odh-ai-gateway-infra(7ed236b2-7d1a-46a6-bbb9-55683420c3ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt" podUID="7ed236b2-7d1a-46a6-bbb9-55683420c3ed" Oct 29 23:30:17.163013 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:17.162966 4821 scope.go:122] "RemoveContainer" containerID="65b707d566a7676188bc60e5c68cced9db6b2e4d5fd1acb1d9692187971ed2c0" Oct 29 23:30:17.163233 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:17.163073 4821 scope.go:122] "RemoveContainer" containerID="34ea57bfec29f84976727255cb6ffde2fc0bc4b544e498bb4df999a0a13d29c6" Oct 29 23:30:17.393384 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:17.393345 4821 generic.go:356] "Generic (PLEG): container finished" podID="bc852c3e-5277-4f30-9063-f4901fdca67e" containerID="699ef04f6f9e0956d846a3892c9f32ba4c7a0df44c4d5754a0b254b1d6b807fc" exitCode=60 Oct 29 23:30:17.393384 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:17.393384 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s" event={"ID":"bc852c3e-5277-4f30-9063-f4901fdca67e","Type":"ContainerDied","Data":"699ef04f6f9e0956d846a3892c9f32ba4c7a0df44c4d5754a0b254b1d6b807fc"} Oct 29 23:30:17.393926 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:17.393439 4821 scope.go:122] "RemoveContainer" containerID="54bd3c27bb143cd2d722c5d525eeb1e42234c2f2f8129377cf2766bd454151e5" Oct 29 23:30:17.393926 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:17.393847 4821 scope.go:122] "RemoveContainer" containerID="699ef04f6f9e0956d846a3892c9f32ba4c7a0df44c4d5754a0b254b1d6b807fc" Oct 29 23:30:17.394244 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:30:17.394194 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s_odh-ai-gateway-infra(bc852c3e-5277-4f30-9063-f4901fdca67e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s" podUID="bc852c3e-5277-4f30-9063-f4901fdca67e" Oct 29 23:30:17.449318 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:17.449287 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt"] Oct 29 23:30:17.681437 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:17.681403 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt" Oct 29 23:30:17.755837 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:17.755804 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed236b2-7d1a-46a6-bbb9-55683420c3ed-kube-api-access-wmjlh\" (UniqueName: \"kubernetes.io/projected/7ed236b2-7d1a-46a6-bbb9-55683420c3ed-kube-api-access-wmjlh\") pod \"7ed236b2-7d1a-46a6-bbb9-55683420c3ed\" (UID: \"7ed236b2-7d1a-46a6-bbb9-55683420c3ed\") " Oct 29 23:30:17.756063 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:17.755884 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ed236b2-7d1a-46a6-bbb9-55683420c3ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed236b2-7d1a-46a6-bbb9-55683420c3ed-maas-api-service-ca\") pod \"7ed236b2-7d1a-46a6-bbb9-55683420c3ed\" (UID: \"7ed236b2-7d1a-46a6-bbb9-55683420c3ed\") " Oct 29 23:30:17.756614 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:17.756505 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed236b2-7d1a-46a6-bbb9-55683420c3ed-maas-api-service-ca" pod "7ed236b2-7d1a-46a6-bbb9-55683420c3ed" (UID: "7ed236b2-7d1a-46a6-bbb9-55683420c3ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:17.758785 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:17.758760 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed236b2-7d1a-46a6-bbb9-55683420c3ed-kube-api-access-wmjlh" pod "7ed236b2-7d1a-46a6-bbb9-55683420c3ed" (UID: "7ed236b2-7d1a-46a6-bbb9-55683420c3ed"). InnerVolumeSpecName "kube-api-access-wmjlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:17.858049 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:17.858009 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmjlh\" (UniqueName: \"kubernetes.io/projected/7ed236b2-7d1a-46a6-bbb9-55683420c3ed-kube-api-access-wmjlh\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:30:17.858049 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:17.858043 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed236b2-7d1a-46a6-bbb9-55683420c3ed-maas-api-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:30:18.405789 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:18.405749 4821 generic.go:356] "Generic (PLEG): container finished" podID="ca5af571-204a-4e17-890f-3a3d929c0827" containerID="c9d3fd021da45a51bab2e2fddd1096ca5eca189bc70db5dc0e4f82fe68e8fd39" exitCode=60 Oct 29 23:30:18.406329 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:18.405825 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l" event={"ID":"ca5af571-204a-4e17-890f-3a3d929c0827","Type":"ContainerDied","Data":"c9d3fd021da45a51bab2e2fddd1096ca5eca189bc70db5dc0e4f82fe68e8fd39"} Oct 29 23:30:18.406329 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:18.405887 4821 scope.go:122] "RemoveContainer" containerID="34ea57bfec29f84976727255cb6ffde2fc0bc4b544e498bb4df999a0a13d29c6" Oct 29 23:30:18.406480 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:18.406376 4821 scope.go:122] "RemoveContainer" containerID="c9d3fd021da45a51bab2e2fddd1096ca5eca189bc70db5dc0e4f82fe68e8fd39" Oct 29 23:30:18.407045 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:30:18.406903 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l_odh-ai-gateway-infra(ca5af571-204a-4e17-890f-3a3d929c0827)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l" podUID="ca5af571-204a-4e17-890f-3a3d929c0827" Oct 29 23:30:18.409825 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:18.409798 4821 generic.go:356] "Generic (PLEG): container finished" podID="99fcc999-240e-4254-9926-d1af30f9f8f8" containerID="874f6dc0abffc7be255aa351dfbe4732f0fc31b39affacb207d645025a955960" exitCode=60 Oct 29 23:30:18.409997 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:18.409831 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf" event={"ID":"99fcc999-240e-4254-9926-d1af30f9f8f8","Type":"ContainerDied","Data":"874f6dc0abffc7be255aa351dfbe4732f0fc31b39affacb207d645025a955960"} Oct 29 23:30:18.410376 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:18.410354 4821 scope.go:122] "RemoveContainer" containerID="874f6dc0abffc7be255aa351dfbe4732f0fc31b39affacb207d645025a955960" Oct 29 23:30:18.410716 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:30:18.410691 4821 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf_odh-ai-gateway-infra(99fcc999-240e-4254-9926-d1af30f9f8f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf" podUID="99fcc999-240e-4254-9926-d1af30f9f8f8" Oct 29 23:30:18.422236 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:18.414293 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt" event={"ID":"7ed236b2-7d1a-46a6-bbb9-55683420c3ed","Type":"ContainerDied","Data":"3d3335fadea9a3a5d09ea7ce8ab643e0170e8028512d125141053f9859d66a5e"} Oct 29 23:30:18.422236 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:18.414332 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt" Oct 29 23:30:18.424103 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:18.424085 4821 scope.go:122] "RemoveContainer" containerID="65b707d566a7676188bc60e5c68cced9db6b2e4d5fd1acb1d9692187971ed2c0" Oct 29 23:30:18.439622 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:18.439595 4821 scope.go:122] "RemoveContainer" containerID="366c910ef659824023df1eacce8b4838d2829e2dcec4ed988a360937e009b2a2" Oct 29 23:30:18.462462 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:18.462433 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s"] Oct 29 23:30:18.544905 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:18.543178 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt"] Oct 29 23:30:18.559591 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:18.558547 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-575d59-29888610-tzqvt"] Oct 29 23:30:18.653262 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:18.653230 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s" Oct 29 23:30:18.816355 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:18.816259 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc852c3e-5277-4f30-9063-f4901fdca67e-kube-api-access-p57m9\" (UniqueName: \"kubernetes.io/projected/bc852c3e-5277-4f30-9063-f4901fdca67e-kube-api-access-p57m9\") pod \"bc852c3e-5277-4f30-9063-f4901fdca67e\" (UID: \"bc852c3e-5277-4f30-9063-f4901fdca67e\") " Oct 29 23:30:18.816355 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:18.816322 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc852c3e-5277-4f30-9063-f4901fdca67e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc852c3e-5277-4f30-9063-f4901fdca67e-maas-api-service-ca\") pod \"bc852c3e-5277-4f30-9063-f4901fdca67e\" (UID: \"bc852c3e-5277-4f30-9063-f4901fdca67e\") " Oct 29 23:30:18.818768 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:18.818731 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc852c3e-5277-4f30-9063-f4901fdca67e-maas-api-service-ca" pod "bc852c3e-5277-4f30-9063-f4901fdca67e" (UID: "bc852c3e-5277-4f30-9063-f4901fdca67e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:18.824261 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:18.824226 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc852c3e-5277-4f30-9063-f4901fdca67e-kube-api-access-p57m9" pod "bc852c3e-5277-4f30-9063-f4901fdca67e" (UID: "bc852c3e-5277-4f30-9063-f4901fdca67e"). InnerVolumeSpecName "kube-api-access-p57m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:18.918663 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:18.918625 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p57m9\" (UniqueName: \"kubernetes.io/projected/bc852c3e-5277-4f30-9063-f4901fdca67e-kube-api-access-p57m9\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:30:18.918663 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:18.918661 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc852c3e-5277-4f30-9063-f4901fdca67e-maas-api-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.439644 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:19.439613 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s" Oct 29 23:30:19.440230 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:19.439938 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s" event={"ID":"bc852c3e-5277-4f30-9063-f4901fdca67e","Type":"ContainerDied","Data":"6a0014d0f4b1d88044362859512b0aa43652bfd2377935f79765b69b92aa7eb9"} Oct 29 23:30:19.440355 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:19.440302 4821 scope.go:122] "RemoveContainer" containerID="699ef04f6f9e0956d846a3892c9f32ba4c7a0df44c4d5754a0b254b1d6b807fc" Oct 29 23:30:19.474601 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:19.474567 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l"] Oct 29 23:30:19.506356 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:19.506304 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s"] Oct 29 23:30:19.512188 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:19.512155 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf"] Oct 29 23:30:19.515998 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:19.515972 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e170ed-29888610-wsf6s"] Oct 29 23:30:19.674877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:19.674811 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l" Oct 29 23:30:19.727255 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:19.727188 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf" Oct 29 23:30:19.733096 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:19.733064 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99fcc999-240e-4254-9926-d1af30f9f8f8-kube-api-access-5mrv5\" (UniqueName: \"kubernetes.io/projected/99fcc999-240e-4254-9926-d1af30f9f8f8-kube-api-access-5mrv5\") pod \"99fcc999-240e-4254-9926-d1af30f9f8f8\" (UID: \"99fcc999-240e-4254-9926-d1af30f9f8f8\") " Oct 29 23:30:19.733260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:19.733152 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99fcc999-240e-4254-9926-d1af30f9f8f8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99fcc999-240e-4254-9926-d1af30f9f8f8-maas-api-service-ca\") pod \"99fcc999-240e-4254-9926-d1af30f9f8f8\" (UID: \"99fcc999-240e-4254-9926-d1af30f9f8f8\") " Oct 29 23:30:19.733385 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:19.733368 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5af571-204a-4e17-890f-3a3d929c0827-kube-api-access-lkd8d\" (UniqueName: \"kubernetes.io/projected/ca5af571-204a-4e17-890f-3a3d929c0827-kube-api-access-lkd8d\") pod \"ca5af571-204a-4e17-890f-3a3d929c0827\" (UID: \"ca5af571-204a-4e17-890f-3a3d929c0827\") " Oct 29 23:30:19.733443 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:19.733406 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca5af571-204a-4e17-890f-3a3d929c0827-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5af571-204a-4e17-890f-3a3d929c0827-maas-api-service-ca\") pod \"ca5af571-204a-4e17-890f-3a3d929c0827\" (UID: \"ca5af571-204a-4e17-890f-3a3d929c0827\") " Oct 29 23:30:19.734252 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:19.734226 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca5af571-204a-4e17-890f-3a3d929c0827-maas-api-service-ca" pod "ca5af571-204a-4e17-890f-3a3d929c0827" (UID: "ca5af571-204a-4e17-890f-3a3d929c0827"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:19.737237 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:19.735090 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99fcc999-240e-4254-9926-d1af30f9f8f8-maas-api-service-ca" pod "99fcc999-240e-4254-9926-d1af30f9f8f8" (UID: "99fcc999-240e-4254-9926-d1af30f9f8f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:19.737641 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:19.737618 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99fcc999-240e-4254-9926-d1af30f9f8f8-kube-api-access-5mrv5" pod "99fcc999-240e-4254-9926-d1af30f9f8f8" (UID: "99fcc999-240e-4254-9926-d1af30f9f8f8"). InnerVolumeSpecName "kube-api-access-5mrv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:19.739069 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:19.739035 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5af571-204a-4e17-890f-3a3d929c0827-kube-api-access-lkd8d" pod "ca5af571-204a-4e17-890f-3a3d929c0827" (UID: "ca5af571-204a-4e17-890f-3a3d929c0827"). InnerVolumeSpecName "kube-api-access-lkd8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:19.835473 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:19.835419 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkd8d\" (UniqueName: \"kubernetes.io/projected/ca5af571-204a-4e17-890f-3a3d929c0827-kube-api-access-lkd8d\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.835473 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:19.835474 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5af571-204a-4e17-890f-3a3d929c0827-maas-api-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.835698 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:19.835493 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mrv5\" (UniqueName: \"kubernetes.io/projected/99fcc999-240e-4254-9926-d1af30f9f8f8-kube-api-access-5mrv5\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:30:19.835698 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:19.835508 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99fcc999-240e-4254-9926-d1af30f9f8f8-maas-api-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:30:20.170745 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:20.170710 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ed236b2-7d1a-46a6-bbb9-55683420c3ed" path="/var/lib/kubelet/pods/7ed236b2-7d1a-46a6-bbb9-55683420c3ed/volumes" Oct 29 23:30:20.171117 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:20.171104 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc852c3e-5277-4f30-9063-f4901fdca67e" path="/var/lib/kubelet/pods/bc852c3e-5277-4f30-9063-f4901fdca67e/volumes" Oct 29 23:30:20.448892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:20.448807 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf" Oct 29 23:30:20.448892 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:20.448827 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf" event={"ID":"99fcc999-240e-4254-9926-d1af30f9f8f8","Type":"ContainerDied","Data":"c894425ce1e0b23ba437123cdfc6aa780ee6f661f1d472c347090f7cdebc8dc9"} Oct 29 23:30:20.449471 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:20.448903 4821 scope.go:122] "RemoveContainer" containerID="874f6dc0abffc7be255aa351dfbe4732f0fc31b39affacb207d645025a955960" Oct 29 23:30:20.451712 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:20.451682 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l" event={"ID":"ca5af571-204a-4e17-890f-3a3d929c0827","Type":"ContainerDied","Data":"299ed0ce014232bdc563c844fab940f57651b7ff9b26a4cc175a34385c23d628"} Oct 29 23:30:20.451823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:20.451696 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l" Oct 29 23:30:20.459467 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:20.459433 4821 scope.go:122] "RemoveContainer" containerID="c9d3fd021da45a51bab2e2fddd1096ca5eca189bc70db5dc0e4f82fe68e8fd39" Oct 29 23:30:20.537650 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:20.537616 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf"] Oct 29 23:30:20.545605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:20.545576 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed3bf2-29888610-bxwrf"] Oct 29 23:30:20.581189 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:20.581164 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l"] Oct 29 23:30:20.587162 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:20.587132 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-59eefa-29888610-4l85l"] Oct 29 23:30:22.176078 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:22.176039 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99fcc999-240e-4254-9926-d1af30f9f8f8" path="/var/lib/kubelet/pods/99fcc999-240e-4254-9926-d1af30f9f8f8/volumes" Oct 29 23:30:22.176719 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:22.176700 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca5af571-204a-4e17-890f-3a3d929c0827" path="/var/lib/kubelet/pods/ca5af571-204a-4e17-890f-3a3d929c0827/volumes" Oct 29 23:30:23.884115 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:30:23.884080 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:30:53.901504 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:30:53.901461 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:30:55.015266 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:55.015232 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s"] Oct 29 23:30:55.015765 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:55.015517 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" podUID="47a79417-347e-49b8-b4b5-45896eccdde2" containerName="maas-api" containerID="cri-o://e6be7b758c0d252e76914a2340bb6714310860a9ea47e47a0ac62fad7e13a597" gracePeriod=30 Oct 29 23:30:55.484581 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:55.484540 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" Oct 29 23:30:55.629059 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:55.629025 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47a79417-347e-49b8-b4b5-45896eccdde2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47a79417-347e-49b8-b4b5-45896eccdde2-maas-api-tls\") pod \"47a79417-347e-49b8-b4b5-45896eccdde2\" (UID: \"47a79417-347e-49b8-b4b5-45896eccdde2\") " Oct 29 23:30:55.629294 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:55.629179 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47a79417-347e-49b8-b4b5-45896eccdde2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47a79417-347e-49b8-b4b5-45896eccdde2-openshift-service-ca\") pod \"47a79417-347e-49b8-b4b5-45896eccdde2\" (UID: \"47a79417-347e-49b8-b4b5-45896eccdde2\") " Oct 29 23:30:55.629294 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:55.629238 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a79417-347e-49b8-b4b5-45896eccdde2-kube-api-access-zndmd\" (UniqueName: \"kubernetes.io/projected/47a79417-347e-49b8-b4b5-45896eccdde2-kube-api-access-zndmd\") pod \"47a79417-347e-49b8-b4b5-45896eccdde2\" (UID: \"47a79417-347e-49b8-b4b5-45896eccdde2\") " Oct 29 23:30:55.629418 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:55.629364 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47a79417-347e-49b8-b4b5-45896eccdde2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/47a79417-347e-49b8-b4b5-45896eccdde2-metrics-certs\") pod \"47a79417-347e-49b8-b4b5-45896eccdde2\" (UID: \"47a79417-347e-49b8-b4b5-45896eccdde2\") " Oct 29 23:30:55.629578 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:55.629554 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47a79417-347e-49b8-b4b5-45896eccdde2-openshift-service-ca" pod "47a79417-347e-49b8-b4b5-45896eccdde2" (UID: "47a79417-347e-49b8-b4b5-45896eccdde2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:30:55.629814 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:55.629787 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47a79417-347e-49b8-b4b5-45896eccdde2-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:30:55.631345 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:55.631320 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47a79417-347e-49b8-b4b5-45896eccdde2-maas-api-tls" pod "47a79417-347e-49b8-b4b5-45896eccdde2" (UID: "47a79417-347e-49b8-b4b5-45896eccdde2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:30:55.631489 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:55.631462 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a79417-347e-49b8-b4b5-45896eccdde2-kube-api-access-zndmd" pod "47a79417-347e-49b8-b4b5-45896eccdde2" (UID: "47a79417-347e-49b8-b4b5-45896eccdde2"). InnerVolumeSpecName "kube-api-access-zndmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:30:55.631588 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:55.631564 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47a79417-347e-49b8-b4b5-45896eccdde2-metrics-certs" pod "47a79417-347e-49b8-b4b5-45896eccdde2" (UID: "47a79417-347e-49b8-b4b5-45896eccdde2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:30:55.728114 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:55.728020 4821 generic.go:356] "Generic (PLEG): container finished" podID="47a79417-347e-49b8-b4b5-45896eccdde2" containerID="e6be7b758c0d252e76914a2340bb6714310860a9ea47e47a0ac62fad7e13a597" exitCode=0 Oct 29 23:30:55.728114 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:55.728067 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" Oct 29 23:30:55.728114 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:55.728101 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" event={"ID":"47a79417-347e-49b8-b4b5-45896eccdde2","Type":"ContainerDied","Data":"e6be7b758c0d252e76914a2340bb6714310860a9ea47e47a0ac62fad7e13a597"} Oct 29 23:30:55.728418 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:55.728157 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" event={"ID":"47a79417-347e-49b8-b4b5-45896eccdde2","Type":"ContainerDied","Data":"8248e8cae6d828dbd6fea38cda8d53916e1bfc5cf220acd72298d3a6e73a2e88"} Oct 29 23:30:55.728418 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:55.728186 4821 scope.go:122] "RemoveContainer" containerID="e6be7b758c0d252e76914a2340bb6714310860a9ea47e47a0ac62fad7e13a597" Oct 29 23:30:55.730453 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:55.730422 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zndmd\" (UniqueName: \"kubernetes.io/projected/47a79417-347e-49b8-b4b5-45896eccdde2-kube-api-access-zndmd\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:30:55.730453 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:55.730447 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47a79417-347e-49b8-b4b5-45896eccdde2-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:30:55.730642 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:55.730460 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47a79417-347e-49b8-b4b5-45896eccdde2-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:30:55.740929 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:55.740913 4821 scope.go:122] "RemoveContainer" containerID="e6be7b758c0d252e76914a2340bb6714310860a9ea47e47a0ac62fad7e13a597" Oct 29 23:30:55.741272 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:30:55.741252 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6be7b758c0d252e76914a2340bb6714310860a9ea47e47a0ac62fad7e13a597\": container with ID starting with e6be7b758c0d252e76914a2340bb6714310860a9ea47e47a0ac62fad7e13a597 not found: ID does not exist" containerID="e6be7b758c0d252e76914a2340bb6714310860a9ea47e47a0ac62fad7e13a597" Oct 29 23:30:55.741359 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:55.741280 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6be7b758c0d252e76914a2340bb6714310860a9ea47e47a0ac62fad7e13a597"} err="failed to get container status \"e6be7b758c0d252e76914a2340bb6714310860a9ea47e47a0ac62fad7e13a597\": rpc error: code = NotFound desc = could not find container \"e6be7b758c0d252e76914a2340bb6714310860a9ea47e47a0ac62fad7e13a597\": container with ID starting with e6be7b758c0d252e76914a2340bb6714310860a9ea47e47a0ac62fad7e13a597 not found: ID does not exist" Oct 29 23:30:55.783015 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:55.782981 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s"] Oct 29 23:30:55.787115 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:55.787092 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s"] Oct 29 23:30:56.170751 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:56.170717 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47a79417-347e-49b8-b4b5-45896eccdde2" path="/var/lib/kubelet/pods/47a79417-347e-49b8-b4b5-45896eccdde2/volumes" Oct 29 23:30:58.203614 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:30:58.203577 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b0c3c3-f56d95d96-g4n2s" podUID="47a79417-347e-49b8-b4b5-45896eccdde2" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.190:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:31:23.884513 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:31:23.884465 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:31:27.936047 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:27.936005 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg"] Oct 29 23:31:27.958749 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:27.958716 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" podUID="0d738ff1-6c39-4e26-a597-fbbe75dd076b" containerName="istio-proxy" containerID="cri-o://8c75f81d1d577b307dad77ad9f7a29e81c43b3fc4cb6e60907e55d66755da172" gracePeriod=30 Oct 29 23:31:28.194165 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:28.194076 4821 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:31:28.194165 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:28.194132 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" podUID="0d738ff1-6c39-4e26-a597-fbbe75dd076b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:31:33.225335 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.225275 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:31:33.336651 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.336620 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-token\" (UniqueName: \"kubernetes.io/projected/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-token\") pod \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " Oct 29 23:31:33.336832 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.336667 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istiod-ca-cert\") pod \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " Oct 29 23:31:33.336832 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.336724 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-workload-socket\") pod \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " Oct 29 23:31:33.336832 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.336744 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-credential-socket\") pod \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " Oct 29 23:31:33.336832 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.336775 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-data\") pod \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " Oct 29 23:31:33.337062 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.336887 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-envoy\") pod \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " Oct 29 23:31:33.337062 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.336979 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-podinfo\") pod \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " Oct 29 23:31:33.337062 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.337039 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-workload-certs\") pod \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " Oct 29 23:31:33.337281 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.337066 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d738ff1-6c39-4e26-a597-fbbe75dd076b-kube-api-access-pkpjr\" (UniqueName: \"kubernetes.io/projected/0d738ff1-6c39-4e26-a597-fbbe75dd076b-kube-api-access-pkpjr\") pod \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\" (UID: \"0d738ff1-6c39-4e26-a597-fbbe75dd076b\") " Oct 29 23:31:33.337281 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.337094 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-credential-socket" pod "0d738ff1-6c39-4e26-a597-fbbe75dd076b" (UID: "0d738ff1-6c39-4e26-a597-fbbe75dd076b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:33.337281 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.337172 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-data" pod "0d738ff1-6c39-4e26-a597-fbbe75dd076b" (UID: "0d738ff1-6c39-4e26-a597-fbbe75dd076b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:33.337281 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.337189 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istiod-ca-cert" pod "0d738ff1-6c39-4e26-a597-fbbe75dd076b" (UID: "0d738ff1-6c39-4e26-a597-fbbe75dd076b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:33.337530 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.337350 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-workload-certs" pod "0d738ff1-6c39-4e26-a597-fbbe75dd076b" (UID: "0d738ff1-6c39-4e26-a597-fbbe75dd076b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:33.337530 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.337450 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-workload-socket" pod "0d738ff1-6c39-4e26-a597-fbbe75dd076b" (UID: "0d738ff1-6c39-4e26-a597-fbbe75dd076b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:33.337637 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.337599 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:31:33.337637 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.337618 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:31:33.337637 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.337633 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:31:33.337821 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.337649 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:31:33.337821 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.337664 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:31:33.339098 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.339074 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-token" pod "0d738ff1-6c39-4e26-a597-fbbe75dd076b" (UID: "0d738ff1-6c39-4e26-a597-fbbe75dd076b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:33.339273 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.339257 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-podinfo" pod "0d738ff1-6c39-4e26-a597-fbbe75dd076b" (UID: "0d738ff1-6c39-4e26-a597-fbbe75dd076b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:31:33.339648 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.339626 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-envoy" pod "0d738ff1-6c39-4e26-a597-fbbe75dd076b" (UID: "0d738ff1-6c39-4e26-a597-fbbe75dd076b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:31:33.339755 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.339703 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d738ff1-6c39-4e26-a597-fbbe75dd076b-kube-api-access-pkpjr" pod "0d738ff1-6c39-4e26-a597-fbbe75dd076b" (UID: "0d738ff1-6c39-4e26-a597-fbbe75dd076b"). InnerVolumeSpecName "kube-api-access-pkpjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:33.439296 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.439262 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:31:33.439296 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.439292 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:31:33.439296 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.439302 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkpjr\" (UniqueName: \"kubernetes.io/projected/0d738ff1-6c39-4e26-a597-fbbe75dd076b-kube-api-access-pkpjr\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:31:33.439560 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:33.439312 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d738ff1-6c39-4e26-a597-fbbe75dd076b-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:31:34.007882 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:34.007847 4821 generic.go:356] "Generic (PLEG): container finished" podID="0d738ff1-6c39-4e26-a597-fbbe75dd076b" containerID="8c75f81d1d577b307dad77ad9f7a29e81c43b3fc4cb6e60907e55d66755da172" exitCode=0 Oct 29 23:31:34.008084 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:34.007899 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" Oct 29 23:31:34.008084 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:34.007933 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" event={"ID":"0d738ff1-6c39-4e26-a597-fbbe75dd076b","Type":"ContainerDied","Data":"8c75f81d1d577b307dad77ad9f7a29e81c43b3fc4cb6e60907e55d66755da172"} Oct 29 23:31:34.008084 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:34.007992 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg" event={"ID":"0d738ff1-6c39-4e26-a597-fbbe75dd076b","Type":"ContainerDied","Data":"b217b632e9c8ce9ba3992d899fadb7f09e95a4a90e33eccc09fc0bd65d178c19"} Oct 29 23:31:34.008084 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:34.008016 4821 scope.go:122] "RemoveContainer" containerID="8c75f81d1d577b307dad77ad9f7a29e81c43b3fc4cb6e60907e55d66755da172" Oct 29 23:31:34.018674 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:34.018655 4821 scope.go:122] "RemoveContainer" containerID="8c75f81d1d577b307dad77ad9f7a29e81c43b3fc4cb6e60907e55d66755da172" Oct 29 23:31:34.018924 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:31:34.018906 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c75f81d1d577b307dad77ad9f7a29e81c43b3fc4cb6e60907e55d66755da172\": container with ID starting with 8c75f81d1d577b307dad77ad9f7a29e81c43b3fc4cb6e60907e55d66755da172 not found: ID does not exist" containerID="8c75f81d1d577b307dad77ad9f7a29e81c43b3fc4cb6e60907e55d66755da172" Oct 29 23:31:34.018974 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:34.018933 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c75f81d1d577b307dad77ad9f7a29e81c43b3fc4cb6e60907e55d66755da172"} err="failed to get container status \"8c75f81d1d577b307dad77ad9f7a29e81c43b3fc4cb6e60907e55d66755da172\": rpc error: code = NotFound desc = could not find container \"8c75f81d1d577b307dad77ad9f7a29e81c43b3fc4cb6e60907e55d66755da172\": container with ID starting with 8c75f81d1d577b307dad77ad9f7a29e81c43b3fc4cb6e60907e55d66755da172 not found: ID does not exist" Oct 29 23:31:34.077255 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:34.077219 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg"] Oct 29 23:31:34.090418 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:34.090392 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ed3bf2-openshift-default-7dc649cdb9-c7vzg"] Oct 29 23:31:34.169496 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:34.169465 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d738ff1-6c39-4e26-a597-fbbe75dd076b" path="/var/lib/kubelet/pods/0d738ff1-6c39-4e26-a597-fbbe75dd076b/volumes" Oct 29 23:31:53.883656 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:31:53.883620 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:31:54.794254 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:54.794217 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs"] Oct 29 23:31:54.794514 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:54.794469 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" podUID="40800b56-bbd2-4274-8031-58667613c72a" containerName="maas-api" containerID="cri-o://f138c4d8fdeacf08c714ac1e8db914883556ffabd154378a7b51a4dfa5929070" gracePeriod=30 Oct 29 23:31:54.949832 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:54.949785 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" podUID="40800b56-bbd2-4274-8031-58667613c72a" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.220:8443/health\": dial tcp 10.133.0.220:8443: connect: connection refused" Oct 29 23:31:55.179110 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:55.178950 4821 generic.go:356] "Generic (PLEG): container finished" podID="40800b56-bbd2-4274-8031-58667613c72a" containerID="f138c4d8fdeacf08c714ac1e8db914883556ffabd154378a7b51a4dfa5929070" exitCode=0 Oct 29 23:31:55.179110 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:55.179043 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" event={"ID":"40800b56-bbd2-4274-8031-58667613c72a","Type":"ContainerDied","Data":"f138c4d8fdeacf08c714ac1e8db914883556ffabd154378a7b51a4dfa5929070"} Oct 29 23:31:55.352078 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:55.352052 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" Oct 29 23:31:55.429849 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:55.429809 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40800b56-bbd2-4274-8031-58667613c72a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40800b56-bbd2-4274-8031-58667613c72a-maas-api-tls\") pod \"40800b56-bbd2-4274-8031-58667613c72a\" (UID: \"40800b56-bbd2-4274-8031-58667613c72a\") " Oct 29 23:31:55.430070 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:55.430047 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40800b56-bbd2-4274-8031-58667613c72a-kube-api-access-qjrn7\" (UniqueName: \"kubernetes.io/projected/40800b56-bbd2-4274-8031-58667613c72a-kube-api-access-qjrn7\") pod \"40800b56-bbd2-4274-8031-58667613c72a\" (UID: \"40800b56-bbd2-4274-8031-58667613c72a\") " Oct 29 23:31:55.430242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:55.430223 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40800b56-bbd2-4274-8031-58667613c72a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/40800b56-bbd2-4274-8031-58667613c72a-metrics-certs\") pod \"40800b56-bbd2-4274-8031-58667613c72a\" (UID: \"40800b56-bbd2-4274-8031-58667613c72a\") " Oct 29 23:31:55.430314 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:55.430279 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40800b56-bbd2-4274-8031-58667613c72a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40800b56-bbd2-4274-8031-58667613c72a-openshift-service-ca\") pod \"40800b56-bbd2-4274-8031-58667613c72a\" (UID: \"40800b56-bbd2-4274-8031-58667613c72a\") " Oct 29 23:31:55.431068 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:55.431039 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40800b56-bbd2-4274-8031-58667613c72a-openshift-service-ca" pod "40800b56-bbd2-4274-8031-58667613c72a" (UID: "40800b56-bbd2-4274-8031-58667613c72a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:31:55.433306 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:55.433281 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40800b56-bbd2-4274-8031-58667613c72a-maas-api-tls" pod "40800b56-bbd2-4274-8031-58667613c72a" (UID: "40800b56-bbd2-4274-8031-58667613c72a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:31:55.433464 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:55.433321 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40800b56-bbd2-4274-8031-58667613c72a-kube-api-access-qjrn7" pod "40800b56-bbd2-4274-8031-58667613c72a" (UID: "40800b56-bbd2-4274-8031-58667613c72a"). InnerVolumeSpecName "kube-api-access-qjrn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:31:55.433579 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:55.433487 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40800b56-bbd2-4274-8031-58667613c72a-metrics-certs" pod "40800b56-bbd2-4274-8031-58667613c72a" (UID: "40800b56-bbd2-4274-8031-58667613c72a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:31:55.532457 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:55.532150 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40800b56-bbd2-4274-8031-58667613c72a-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:31:55.532828 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:55.532643 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40800b56-bbd2-4274-8031-58667613c72a-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:31:55.532828 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:55.532668 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40800b56-bbd2-4274-8031-58667613c72a-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:31:55.532828 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:55.532782 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjrn7\" (UniqueName: \"kubernetes.io/projected/40800b56-bbd2-4274-8031-58667613c72a-kube-api-access-qjrn7\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:31:56.185419 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:56.185387 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" Oct 29 23:31:56.185801 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:56.185436 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs" event={"ID":"40800b56-bbd2-4274-8031-58667613c72a","Type":"ContainerDied","Data":"0434ed809cc903182428208fe73a95df3f46a336189716c4992ff72a73c01f8a"} Oct 29 23:31:56.185801 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:56.185489 4821 scope.go:122] "RemoveContainer" containerID="f138c4d8fdeacf08c714ac1e8db914883556ffabd154378a7b51a4dfa5929070" Oct 29 23:31:56.240875 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:56.240846 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs"] Oct 29 23:31:56.253881 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:56.253857 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-59eefa-75cf7d7584-wr7hs"] Oct 29 23:31:58.168622 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:31:58.168588 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40800b56-bbd2-4274-8031-58667613c72a" path="/var/lib/kubelet/pods/40800b56-bbd2-4274-8031-58667613c72a/volumes" Oct 29 23:32:14.608446 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:14.608408 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs"] Oct 29 23:32:19.608923 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:19.608878 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs" podUID="65b14de3-46e0-438e-9e72-68ab3c19f4b3" containerName="payload-processing" containerID="cri-o://976f9678598c769e51d94d685577369ba7c0b72c72bda7cbb013fe7424ca6ff1" gracePeriod=25 Oct 29 23:32:23.884393 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:32:23.884357 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:32:25.702289 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:25.702249 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z"] Oct 29 23:32:25.702920 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:25.702502 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z" podUID="bd770541-8afc-468c-bdcb-f95ec581d689" containerName="maas-api" containerID="cri-o://1b08cd8a594a96ba7a9ecd1f7dec8b2dfda8a0475d55d58ab769c16357334539" gracePeriod=30 Oct 29 23:32:26.081226 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:26.078367 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z" Oct 29 23:32:26.196676 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:26.196642 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd770541-8afc-468c-bdcb-f95ec581d689-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd770541-8afc-468c-bdcb-f95ec581d689-maas-api-tls\") pod \"bd770541-8afc-468c-bdcb-f95ec581d689\" (UID: \"bd770541-8afc-468c-bdcb-f95ec581d689\") " Oct 29 23:32:26.196842 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:26.196730 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd770541-8afc-468c-bdcb-f95ec581d689-kube-api-access-64xrs\" (UniqueName: \"kubernetes.io/projected/bd770541-8afc-468c-bdcb-f95ec581d689-kube-api-access-64xrs\") pod \"bd770541-8afc-468c-bdcb-f95ec581d689\" (UID: \"bd770541-8afc-468c-bdcb-f95ec581d689\") " Oct 29 23:32:26.196842 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:26.196773 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd770541-8afc-468c-bdcb-f95ec581d689-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd770541-8afc-468c-bdcb-f95ec581d689-metrics-certs\") pod \"bd770541-8afc-468c-bdcb-f95ec581d689\" (UID: \"bd770541-8afc-468c-bdcb-f95ec581d689\") " Oct 29 23:32:26.196842 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:26.196820 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd770541-8afc-468c-bdcb-f95ec581d689-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd770541-8afc-468c-bdcb-f95ec581d689-openshift-service-ca\") pod \"bd770541-8afc-468c-bdcb-f95ec581d689\" (UID: \"bd770541-8afc-468c-bdcb-f95ec581d689\") " Oct 29 23:32:26.197303 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:26.197272 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd770541-8afc-468c-bdcb-f95ec581d689-openshift-service-ca" pod "bd770541-8afc-468c-bdcb-f95ec581d689" (UID: "bd770541-8afc-468c-bdcb-f95ec581d689"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:26.199091 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:26.199060 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd770541-8afc-468c-bdcb-f95ec581d689-metrics-certs" pod "bd770541-8afc-468c-bdcb-f95ec581d689" (UID: "bd770541-8afc-468c-bdcb-f95ec581d689"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:32:26.199261 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:26.199148 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd770541-8afc-468c-bdcb-f95ec581d689-maas-api-tls" pod "bd770541-8afc-468c-bdcb-f95ec581d689" (UID: "bd770541-8afc-468c-bdcb-f95ec581d689"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:32:26.199324 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:26.199256 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd770541-8afc-468c-bdcb-f95ec581d689-kube-api-access-64xrs" pod "bd770541-8afc-468c-bdcb-f95ec581d689" (UID: "bd770541-8afc-468c-bdcb-f95ec581d689"). InnerVolumeSpecName "kube-api-access-64xrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:26.298448 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:26.298369 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd770541-8afc-468c-bdcb-f95ec581d689-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:26.298448 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:26.298399 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64xrs\" (UniqueName: \"kubernetes.io/projected/bd770541-8afc-468c-bdcb-f95ec581d689-kube-api-access-64xrs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:26.298448 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:26.298412 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd770541-8afc-468c-bdcb-f95ec581d689-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:26.298448 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:26.298420 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd770541-8afc-468c-bdcb-f95ec581d689-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:26.380424 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:26.380383 4821 generic.go:356] "Generic (PLEG): container finished" podID="bd770541-8afc-468c-bdcb-f95ec581d689" containerID="1b08cd8a594a96ba7a9ecd1f7dec8b2dfda8a0475d55d58ab769c16357334539" exitCode=0 Oct 29 23:32:26.380647 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:26.380431 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z" Oct 29 23:32:26.380647 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:26.380463 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z" event={"ID":"bd770541-8afc-468c-bdcb-f95ec581d689","Type":"ContainerDied","Data":"1b08cd8a594a96ba7a9ecd1f7dec8b2dfda8a0475d55d58ab769c16357334539"} Oct 29 23:32:26.380647 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:26.380517 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z" event={"ID":"bd770541-8afc-468c-bdcb-f95ec581d689","Type":"ContainerDied","Data":"d436533c58141e4f20e7e713942decc44a20272f25ff96562314d22ce7bfd390"} Oct 29 23:32:26.380647 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:26.380539 4821 scope.go:122] "RemoveContainer" containerID="1b08cd8a594a96ba7a9ecd1f7dec8b2dfda8a0475d55d58ab769c16357334539" Oct 29 23:32:26.391459 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:26.391423 4821 scope.go:122] "RemoveContainer" containerID="1b08cd8a594a96ba7a9ecd1f7dec8b2dfda8a0475d55d58ab769c16357334539" Oct 29 23:32:26.391733 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:32:26.391711 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b08cd8a594a96ba7a9ecd1f7dec8b2dfda8a0475d55d58ab769c16357334539\": container with ID starting with 1b08cd8a594a96ba7a9ecd1f7dec8b2dfda8a0475d55d58ab769c16357334539 not found: ID does not exist" containerID="1b08cd8a594a96ba7a9ecd1f7dec8b2dfda8a0475d55d58ab769c16357334539" Oct 29 23:32:26.391819 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:26.391745 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b08cd8a594a96ba7a9ecd1f7dec8b2dfda8a0475d55d58ab769c16357334539"} err="failed to get container status \"1b08cd8a594a96ba7a9ecd1f7dec8b2dfda8a0475d55d58ab769c16357334539\": rpc error: code = NotFound desc = could not find container \"1b08cd8a594a96ba7a9ecd1f7dec8b2dfda8a0475d55d58ab769c16357334539\": container with ID starting with 1b08cd8a594a96ba7a9ecd1f7dec8b2dfda8a0475d55d58ab769c16357334539 not found: ID does not exist" Oct 29 23:32:26.434694 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:26.434659 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z"] Oct 29 23:32:26.455454 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:26.455394 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e170ed-68568fc97b-48d4z"] Oct 29 23:32:27.311883 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:27.311845 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs" podUID="65b14de3-46e0-438e-9e72-68ab3c19f4b3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:32:28.180971 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:28.180930 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd770541-8afc-468c-bdcb-f95ec581d689" path="/var/lib/kubelet/pods/bd770541-8afc-468c-bdcb-f95ec581d689/volumes" Oct 29 23:32:34.180147 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:34.180115 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh"] Oct 29 23:32:34.180678 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:34.180349 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" podUID="36ea1460-ac2f-410a-b06a-b18f1f8e56d3" containerName="istio-proxy" containerID="cri-o://53156b884a4309f21bd321f88c0fc944f19c04abf6df11ae89d29f3177f85736" gracePeriod=30 Oct 29 23:32:36.107787 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.107749 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf"] Oct 29 23:32:36.108260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108161 4821 cpu_manager.go:414] "containerMap: removing container" podUID="bd770541-8afc-468c-bdcb-f95ec581d689" containerName="maas-api" Oct 29 23:32:36.108260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108172 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd770541-8afc-468c-bdcb-f95ec581d689" containerName="maas-api" Oct 29 23:32:36.108260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108182 4821 cpu_manager.go:414] "containerMap: removing container" podUID="7ed236b2-7d1a-46a6-bbb9-55683420c3ed" containerName="cleanup" Oct 29 23:32:36.108260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108187 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed236b2-7d1a-46a6-bbb9-55683420c3ed" containerName="cleanup" Oct 29 23:32:36.108260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108194 4821 cpu_manager.go:414] "containerMap: removing container" podUID="0d738ff1-6c39-4e26-a597-fbbe75dd076b" containerName="istio-proxy" Oct 29 23:32:36.108260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108219 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d738ff1-6c39-4e26-a597-fbbe75dd076b" containerName="istio-proxy" Oct 29 23:32:36.108260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108233 4821 cpu_manager.go:414] "containerMap: removing container" podUID="40800b56-bbd2-4274-8031-58667613c72a" containerName="maas-api" Oct 29 23:32:36.108260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108238 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40800b56-bbd2-4274-8031-58667613c72a" containerName="maas-api" Oct 29 23:32:36.108260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108246 4821 cpu_manager.go:414] "containerMap: removing container" podUID="47a79417-347e-49b8-b4b5-45896eccdde2" containerName="maas-api" Oct 29 23:32:36.108260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108251 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47a79417-347e-49b8-b4b5-45896eccdde2" containerName="maas-api" Oct 29 23:32:36.108260 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108265 4821 cpu_manager.go:414] "containerMap: removing container" podUID="99fcc999-240e-4254-9926-d1af30f9f8f8" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108270 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99fcc999-240e-4254-9926-d1af30f9f8f8" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108276 4821 cpu_manager.go:414] "containerMap: removing container" podUID="bc852c3e-5277-4f30-9063-f4901fdca67e" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108280 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc852c3e-5277-4f30-9063-f4901fdca67e" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108286 4821 cpu_manager.go:414] "containerMap: removing container" podUID="bc852c3e-5277-4f30-9063-f4901fdca67e" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108290 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc852c3e-5277-4f30-9063-f4901fdca67e" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108298 4821 cpu_manager.go:414] "containerMap: removing container" podUID="ca5af571-204a-4e17-890f-3a3d929c0827" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108302 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5af571-204a-4e17-890f-3a3d929c0827" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108309 4821 cpu_manager.go:414] "containerMap: removing container" podUID="99fcc999-240e-4254-9926-d1af30f9f8f8" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108313 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99fcc999-240e-4254-9926-d1af30f9f8f8" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108318 4821 cpu_manager.go:414] "containerMap: removing container" podUID="ca5af571-204a-4e17-890f-3a3d929c0827" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108322 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5af571-204a-4e17-890f-3a3d929c0827" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108328 4821 cpu_manager.go:414] "containerMap: removing container" podUID="ca5af571-204a-4e17-890f-3a3d929c0827" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108332 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5af571-204a-4e17-890f-3a3d929c0827" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108346 4821 cpu_manager.go:414] "containerMap: removing container" podUID="7ed236b2-7d1a-46a6-bbb9-55683420c3ed" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108350 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed236b2-7d1a-46a6-bbb9-55683420c3ed" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108358 4821 cpu_manager.go:414] "containerMap: removing container" podUID="99fcc999-240e-4254-9926-d1af30f9f8f8" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108362 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99fcc999-240e-4254-9926-d1af30f9f8f8" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108422 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc852c3e-5277-4f30-9063-f4901fdca67e" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108431 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="99fcc999-240e-4254-9926-d1af30f9f8f8" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108439 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd770541-8afc-468c-bdcb-f95ec581d689" containerName="maas-api" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108443 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed236b2-7d1a-46a6-bbb9-55683420c3ed" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108448 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d738ff1-6c39-4e26-a597-fbbe75dd076b" containerName="istio-proxy" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108455 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="40800b56-bbd2-4274-8031-58667613c72a" containerName="maas-api" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108460 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="47a79417-347e-49b8-b4b5-45896eccdde2" containerName="maas-api" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108467 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed236b2-7d1a-46a6-bbb9-55683420c3ed" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108473 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc852c3e-5277-4f30-9063-f4901fdca67e" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108478 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5af571-204a-4e17-890f-3a3d929c0827" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108483 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="99fcc999-240e-4254-9926-d1af30f9f8f8" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108487 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5af571-204a-4e17-890f-3a3d929c0827" containerName="cleanup" Oct 29 23:32:36.108633 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.108492 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5af571-204a-4e17-890f-3a3d929c0827" containerName="cleanup" Oct 29 23:32:36.113000 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.112982 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf" Oct 29 23:32:36.131508 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.131480 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf"] Oct 29 23:32:36.141356 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.141329 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:32:36.141508 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.141386 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:32:36.219955 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.219919 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lsbz\" (UniqueName: \"kubernetes.io/projected/cafaae7c-1ed2-40de-949a-418034ce834f-kube-api-access-2lsbz\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf\" (UID: \"cafaae7c-1ed2-40de-949a-418034ce834f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf" Oct 29 23:32:36.220215 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.220171 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafaae7c-1ed2-40de-949a-418034ce834f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf\" (UID: \"cafaae7c-1ed2-40de-949a-418034ce834f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf" Oct 29 23:32:36.220364 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.220270 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafaae7c-1ed2-40de-949a-418034ce834f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf\" (UID: \"cafaae7c-1ed2-40de-949a-418034ce834f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf" Oct 29 23:32:36.321629 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.321594 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lsbz\" (UniqueName: \"kubernetes.io/projected/cafaae7c-1ed2-40de-949a-418034ce834f-kube-api-access-2lsbz\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf\" (UID: \"cafaae7c-1ed2-40de-949a-418034ce834f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf" Oct 29 23:32:36.321810 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.321683 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafaae7c-1ed2-40de-949a-418034ce834f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf\" (UID: \"cafaae7c-1ed2-40de-949a-418034ce834f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf" Oct 29 23:32:36.321810 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.321708 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafaae7c-1ed2-40de-949a-418034ce834f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf\" (UID: \"cafaae7c-1ed2-40de-949a-418034ce834f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf" Oct 29 23:32:36.322432 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.322404 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafaae7c-1ed2-40de-949a-418034ce834f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf\" (UID: \"cafaae7c-1ed2-40de-949a-418034ce834f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf" Oct 29 23:32:36.324175 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.324154 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafaae7c-1ed2-40de-949a-418034ce834f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf\" (UID: \"cafaae7c-1ed2-40de-949a-418034ce834f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf" Oct 29 23:32:36.346050 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.346022 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lsbz\" (UniqueName: \"kubernetes.io/projected/cafaae7c-1ed2-40de-949a-418034ce834f-kube-api-access-2lsbz\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf\" (UID: \"cafaae7c-1ed2-40de-949a-418034ce834f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf" Oct 29 23:32:36.425963 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.425885 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf" Oct 29 23:32:36.726594 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:36.726383 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf"] Oct 29 23:32:36.729262 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:32:36.729234 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcafaae7c_1ed2_40de_949a_418034ce834f.slice/crio-f0e8a24cfca11dd24a12fde2ec7d1690978f43691cc4427d747573374ab4d9ba WatchSource:0}: Error finding container f0e8a24cfca11dd24a12fde2ec7d1690978f43691cc4427d747573374ab4d9ba: Status 404 returned error can't find the container with id f0e8a24cfca11dd24a12fde2ec7d1690978f43691cc4427d747573374ab4d9ba Oct 29 23:32:37.313073 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:37.312985 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs" podUID="65b14de3-46e0-438e-9e72-68ab3c19f4b3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:32:37.506035 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:37.505946 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf" event={"ID":"cafaae7c-1ed2-40de-949a-418034ce834f","Type":"ContainerStarted","Data":"ff2a5f682785ddf60a471f1b4c17bcb390ce14b8dac8e8cbd019997e27cff7c2"} Oct 29 23:32:37.506035 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:37.506004 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf" event={"ID":"cafaae7c-1ed2-40de-949a-418034ce834f","Type":"ContainerStarted","Data":"f0e8a24cfca11dd24a12fde2ec7d1690978f43691cc4427d747573374ab4d9ba"} Oct 29 23:32:37.547419 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:37.547364 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf" podStartSLOduration=1.547349735 podStartE2EDuration="1.547349735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:37.545749568 +0000 UTC m=+3011.486297477" watchObservedRunningTime="2026-10-29 23:32:37.547349735 +0000 UTC m=+3011.487897643" Oct 29 23:32:38.513537 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:38.513500 4821 generic.go:356] "Generic (PLEG): container finished" podID="cafaae7c-1ed2-40de-949a-418034ce834f" containerID="ff2a5f682785ddf60a471f1b4c17bcb390ce14b8dac8e8cbd019997e27cff7c2" exitCode=0 Oct 29 23:32:38.513950 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:38.513580 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf" event={"ID":"cafaae7c-1ed2-40de-949a-418034ce834f","Type":"ContainerDied","Data":"ff2a5f682785ddf60a471f1b4c17bcb390ce14b8dac8e8cbd019997e27cff7c2"} Oct 29 23:32:39.443085 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.443024 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:32:39.521726 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.521689 4821 generic.go:356] "Generic (PLEG): container finished" podID="36ea1460-ac2f-410a-b06a-b18f1f8e56d3" containerID="53156b884a4309f21bd321f88c0fc944f19c04abf6df11ae89d29f3177f85736" exitCode=0 Oct 29 23:32:39.522252 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.521742 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" Oct 29 23:32:39.522252 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.521776 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" event={"ID":"36ea1460-ac2f-410a-b06a-b18f1f8e56d3","Type":"ContainerDied","Data":"53156b884a4309f21bd321f88c0fc944f19c04abf6df11ae89d29f3177f85736"} Oct 29 23:32:39.522252 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.521841 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh" event={"ID":"36ea1460-ac2f-410a-b06a-b18f1f8e56d3","Type":"ContainerDied","Data":"0879c13d04b735328dda1e6fe99c1e104cd87dff21eac5a018cfafe90ecb714c"} Oct 29 23:32:39.522252 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.521875 4821 scope.go:122] "RemoveContainer" containerID="53156b884a4309f21bd321f88c0fc944f19c04abf6df11ae89d29f3177f85736" Oct 29 23:32:39.533541 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.533524 4821 scope.go:122] "RemoveContainer" containerID="528e90721b9ce1aad59011a013291ebbf49717ddb34cf6e3b7c39895fb10beb7" Oct 29 23:32:39.544194 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.543945 4821 scope.go:122] "RemoveContainer" containerID="53156b884a4309f21bd321f88c0fc944f19c04abf6df11ae89d29f3177f85736" Oct 29 23:32:39.544366 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:32:39.544341 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53156b884a4309f21bd321f88c0fc944f19c04abf6df11ae89d29f3177f85736\": container with ID starting with 53156b884a4309f21bd321f88c0fc944f19c04abf6df11ae89d29f3177f85736 not found: ID does not exist" containerID="53156b884a4309f21bd321f88c0fc944f19c04abf6df11ae89d29f3177f85736" Oct 29 23:32:39.544422 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.544377 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53156b884a4309f21bd321f88c0fc944f19c04abf6df11ae89d29f3177f85736"} err="failed to get container status \"53156b884a4309f21bd321f88c0fc944f19c04abf6df11ae89d29f3177f85736\": rpc error: code = NotFound desc = could not find container \"53156b884a4309f21bd321f88c0fc944f19c04abf6df11ae89d29f3177f85736\": container with ID starting with 53156b884a4309f21bd321f88c0fc944f19c04abf6df11ae89d29f3177f85736 not found: ID does not exist" Oct 29 23:32:39.544422 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.544395 4821 scope.go:122] "RemoveContainer" containerID="528e90721b9ce1aad59011a013291ebbf49717ddb34cf6e3b7c39895fb10beb7" Oct 29 23:32:39.544694 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:32:39.544675 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"528e90721b9ce1aad59011a013291ebbf49717ddb34cf6e3b7c39895fb10beb7\": container with ID starting with 528e90721b9ce1aad59011a013291ebbf49717ddb34cf6e3b7c39895fb10beb7 not found: ID does not exist" containerID="528e90721b9ce1aad59011a013291ebbf49717ddb34cf6e3b7c39895fb10beb7" Oct 29 23:32:39.544786 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.544704 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528e90721b9ce1aad59011a013291ebbf49717ddb34cf6e3b7c39895fb10beb7"} err="failed to get container status \"528e90721b9ce1aad59011a013291ebbf49717ddb34cf6e3b7c39895fb10beb7\": rpc error: code = NotFound desc = could not find container \"528e90721b9ce1aad59011a013291ebbf49717ddb34cf6e3b7c39895fb10beb7\": container with ID starting with 528e90721b9ce1aad59011a013291ebbf49717ddb34cf6e3b7c39895fb10beb7 not found: ID does not exist" Oct 29 23:32:39.566121 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.566092 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-credential-socket\") pod \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " Oct 29 23:32:39.566317 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.566146 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-workload-socket\") pod \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " Oct 29 23:32:39.566317 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.566257 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-token\" (UniqueName: \"kubernetes.io/projected/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-token\") pod \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " Oct 29 23:32:39.566317 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.566301 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-kube-api-access-4vm57\" (UniqueName: \"kubernetes.io/projected/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-kube-api-access-4vm57\") pod \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " Oct 29 23:32:39.566596 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.566333 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-podinfo\") pod \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " Oct 29 23:32:39.566596 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.566386 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-data\") pod \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " Oct 29 23:32:39.566596 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.566446 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-envoy\") pod \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " Oct 29 23:32:39.566596 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.566493 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istiod-ca-cert\") pod \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " Oct 29 23:32:39.566596 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.566499 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-credential-socket" pod "36ea1460-ac2f-410a-b06a-b18f1f8e56d3" (UID: "36ea1460-ac2f-410a-b06a-b18f1f8e56d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:39.566596 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.566527 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-workload-certs\") pod \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\" (UID: \"36ea1460-ac2f-410a-b06a-b18f1f8e56d3\") " Oct 29 23:32:39.566937 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.566628 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-workload-socket" pod "36ea1460-ac2f-410a-b06a-b18f1f8e56d3" (UID: "36ea1460-ac2f-410a-b06a-b18f1f8e56d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:39.566997 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.566956 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:39.566997 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.566978 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:39.566997 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.566984 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-data" pod "36ea1460-ac2f-410a-b06a-b18f1f8e56d3" (UID: "36ea1460-ac2f-410a-b06a-b18f1f8e56d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:39.567148 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.567049 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istiod-ca-cert" pod "36ea1460-ac2f-410a-b06a-b18f1f8e56d3" (UID: "36ea1460-ac2f-410a-b06a-b18f1f8e56d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:39.567561 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.567534 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-workload-certs" pod "36ea1460-ac2f-410a-b06a-b18f1f8e56d3" (UID: "36ea1460-ac2f-410a-b06a-b18f1f8e56d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:39.569450 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.569423 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-envoy" pod "36ea1460-ac2f-410a-b06a-b18f1f8e56d3" (UID: "36ea1460-ac2f-410a-b06a-b18f1f8e56d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:39.569616 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.569543 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-token" pod "36ea1460-ac2f-410a-b06a-b18f1f8e56d3" (UID: "36ea1460-ac2f-410a-b06a-b18f1f8e56d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:39.569838 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.569819 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-podinfo" pod "36ea1460-ac2f-410a-b06a-b18f1f8e56d3" (UID: "36ea1460-ac2f-410a-b06a-b18f1f8e56d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:32:39.569918 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.569877 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-kube-api-access-4vm57" pod "36ea1460-ac2f-410a-b06a-b18f1f8e56d3" (UID: "36ea1460-ac2f-410a-b06a-b18f1f8e56d3"). InnerVolumeSpecName "kube-api-access-4vm57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:39.667691 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.667663 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:39.667691 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.667694 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:39.667885 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.667707 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:39.667885 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.667716 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:39.667885 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.667724 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:39.667885 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.667731 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vm57\" (UniqueName: \"kubernetes.io/projected/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-kube-api-access-4vm57\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:39.667885 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.667741 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ea1460-ac2f-410a-b06a-b18f1f8e56d3-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:39.695966 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.695900 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v"] Oct 29 23:32:39.722328 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.722318 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf" Oct 29 23:32:39.746971 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.746939 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w"] Oct 29 23:32:39.832115 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.832084 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw"] Oct 29 23:32:39.832347 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.832318 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" podUID="77d46565-e43f-44a8-8d78-4417e7016c5d" containerName="istio-proxy" containerID="cri-o://2147d171319ef1ed54a31f87fa4113cc09f04fdff294d37bee328d4640bf3e07" gracePeriod=30 Oct 29 23:32:39.869271 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.869235 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cafaae7c-1ed2-40de-949a-418034ce834f-kube-api-access-2lsbz\" (UniqueName: \"kubernetes.io/projected/cafaae7c-1ed2-40de-949a-418034ce834f-kube-api-access-2lsbz\") pod \"cafaae7c-1ed2-40de-949a-418034ce834f\" (UID: \"cafaae7c-1ed2-40de-949a-418034ce834f\") " Oct 29 23:32:39.869462 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.869334 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cafaae7c-1ed2-40de-949a-418034ce834f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafaae7c-1ed2-40de-949a-418034ce834f-maas-api-openshift-ca\") pod \"cafaae7c-1ed2-40de-949a-418034ce834f\" (UID: \"cafaae7c-1ed2-40de-949a-418034ce834f\") " Oct 29 23:32:39.870063 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.869830 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cafaae7c-1ed2-40de-949a-418034ce834f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafaae7c-1ed2-40de-949a-418034ce834f-maas-api-xks-ca\") pod \"cafaae7c-1ed2-40de-949a-418034ce834f\" (UID: \"cafaae7c-1ed2-40de-949a-418034ce834f\") " Oct 29 23:32:39.870063 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.869884 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cafaae7c-1ed2-40de-949a-418034ce834f-maas-api-openshift-ca" pod "cafaae7c-1ed2-40de-949a-418034ce834f" (UID: "cafaae7c-1ed2-40de-949a-418034ce834f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:39.870571 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.870311 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafaae7c-1ed2-40de-949a-418034ce834f-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:39.871756 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.871732 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafaae7c-1ed2-40de-949a-418034ce834f-kube-api-access-2lsbz" pod "cafaae7c-1ed2-40de-949a-418034ce834f" (UID: "cafaae7c-1ed2-40de-949a-418034ce834f"). InnerVolumeSpecName "kube-api-access-2lsbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:39.872158 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.872127 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cafaae7c-1ed2-40de-949a-418034ce834f-maas-api-xks-ca" pod "cafaae7c-1ed2-40de-949a-418034ce834f" (UID: "cafaae7c-1ed2-40de-949a-418034ce834f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:32:39.884404 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.884368 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs" Oct 29 23:32:39.971156 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.971077 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65b14de3-46e0-438e-9e72-68ab3c19f4b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b14de3-46e0-438e-9e72-68ab3c19f4b3-plugins-config-volume\") pod \"65b14de3-46e0-438e-9e72-68ab3c19f4b3\" (UID: \"65b14de3-46e0-438e-9e72-68ab3c19f4b3\") " Oct 29 23:32:39.971360 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.971181 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65b14de3-46e0-438e-9e72-68ab3c19f4b3-kube-api-access-zp7v9\" (UniqueName: \"kubernetes.io/projected/65b14de3-46e0-438e-9e72-68ab3c19f4b3-kube-api-access-zp7v9\") pod \"65b14de3-46e0-438e-9e72-68ab3c19f4b3\" (UID: \"65b14de3-46e0-438e-9e72-68ab3c19f4b3\") " Oct 29 23:32:39.971582 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.971558 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65b14de3-46e0-438e-9e72-68ab3c19f4b3-plugins-config-volume" pod "65b14de3-46e0-438e-9e72-68ab3c19f4b3" (UID: "65b14de3-46e0-438e-9e72-68ab3c19f4b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:39.971651 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.971580 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lsbz\" (UniqueName: \"kubernetes.io/projected/cafaae7c-1ed2-40de-949a-418034ce834f-kube-api-access-2lsbz\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:39.971651 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.971601 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafaae7c-1ed2-40de-949a-418034ce834f-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:39.973558 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:39.973531 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b14de3-46e0-438e-9e72-68ab3c19f4b3-kube-api-access-zp7v9" pod "65b14de3-46e0-438e-9e72-68ab3c19f4b3" (UID: "65b14de3-46e0-438e-9e72-68ab3c19f4b3"). InnerVolumeSpecName "kube-api-access-zp7v9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:40.073013 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:40.072980 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b14de3-46e0-438e-9e72-68ab3c19f4b3-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:40.073013 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:40.073009 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp7v9\" (UniqueName: \"kubernetes.io/projected/65b14de3-46e0-438e-9e72-68ab3c19f4b3-kube-api-access-zp7v9\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:40.109976 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:40.109937 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh"] Oct 29 23:32:40.116302 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:40.116276 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e170ed-openshift-default-6d7b6644ff-tn2dh"] Oct 29 23:32:40.169064 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:40.169035 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36ea1460-ac2f-410a-b06a-b18f1f8e56d3" path="/var/lib/kubelet/pods/36ea1460-ac2f-410a-b06a-b18f1f8e56d3/volumes" Oct 29 23:32:40.529567 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:40.529528 4821 generic.go:356] "Generic (PLEG): container finished" podID="65b14de3-46e0-438e-9e72-68ab3c19f4b3" containerID="976f9678598c769e51d94d685577369ba7c0b72c72bda7cbb013fe7424ca6ff1" exitCode=0 Oct 29 23:32:40.530122 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:40.529578 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs" Oct 29 23:32:40.530122 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:40.529610 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs" event={"ID":"65b14de3-46e0-438e-9e72-68ab3c19f4b3","Type":"ContainerDied","Data":"976f9678598c769e51d94d685577369ba7c0b72c72bda7cbb013fe7424ca6ff1"} Oct 29 23:32:40.530122 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:40.529664 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs" event={"ID":"65b14de3-46e0-438e-9e72-68ab3c19f4b3","Type":"ContainerDied","Data":"089e9f32652d1c18dd28d130e9795588da703f295a2ed711557f1c516cbd6ce5"} Oct 29 23:32:40.530122 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:40.529690 4821 scope.go:122] "RemoveContainer" containerID="976f9678598c769e51d94d685577369ba7c0b72c72bda7cbb013fe7424ca6ff1" Oct 29 23:32:40.531232 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:40.531172 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf" event={"ID":"cafaae7c-1ed2-40de-949a-418034ce834f","Type":"ContainerDied","Data":"f0e8a24cfca11dd24a12fde2ec7d1690978f43691cc4427d747573374ab4d9ba"} Oct 29 23:32:40.531329 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:40.531232 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf" Oct 29 23:32:40.531329 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:40.531255 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0e8a24cfca11dd24a12fde2ec7d1690978f43691cc4427d747573374ab4d9ba" Oct 29 23:32:40.539830 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:40.539807 4821 scope.go:122] "RemoveContainer" containerID="976f9678598c769e51d94d685577369ba7c0b72c72bda7cbb013fe7424ca6ff1" Oct 29 23:32:40.540119 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:32:40.540096 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"976f9678598c769e51d94d685577369ba7c0b72c72bda7cbb013fe7424ca6ff1\": container with ID starting with 976f9678598c769e51d94d685577369ba7c0b72c72bda7cbb013fe7424ca6ff1 not found: ID does not exist" containerID="976f9678598c769e51d94d685577369ba7c0b72c72bda7cbb013fe7424ca6ff1" Oct 29 23:32:40.540179 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:40.540131 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"976f9678598c769e51d94d685577369ba7c0b72c72bda7cbb013fe7424ca6ff1"} err="failed to get container status \"976f9678598c769e51d94d685577369ba7c0b72c72bda7cbb013fe7424ca6ff1\": rpc error: code = NotFound desc = could not find container \"976f9678598c769e51d94d685577369ba7c0b72c72bda7cbb013fe7424ca6ff1\": container with ID starting with 976f9678598c769e51d94d685577369ba7c0b72c72bda7cbb013fe7424ca6ff1 not found: ID does not exist" Oct 29 23:32:40.601679 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:40.601644 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs"] Oct 29 23:32:40.603940 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:40.603918 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-e170ed-f7999fcbd-l67hs"] Oct 29 23:32:42.174613 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:42.174458 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65b14de3-46e0-438e-9e72-68ab3c19f4b3" path="/var/lib/kubelet/pods/65b14de3-46e0-438e-9e72-68ab3c19f4b3/volumes" Oct 29 23:32:45.105330 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.105271 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:32:45.228941 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.228908 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-podinfo\") pod \"77d46565-e43f-44a8-8d78-4417e7016c5d\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " Oct 29 23:32:45.229132 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.228978 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-envoy\") pod \"77d46565-e43f-44a8-8d78-4417e7016c5d\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " Oct 29 23:32:45.229132 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.229013 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-workload-certs\") pod \"77d46565-e43f-44a8-8d78-4417e7016c5d\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " Oct 29 23:32:45.229132 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.229032 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-data\") pod \"77d46565-e43f-44a8-8d78-4417e7016c5d\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " Oct 29 23:32:45.229132 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.229050 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-workload-socket\") pod \"77d46565-e43f-44a8-8d78-4417e7016c5d\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " Oct 29 23:32:45.229132 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.229075 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-token\" (UniqueName: \"kubernetes.io/projected/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-token\") pod \"77d46565-e43f-44a8-8d78-4417e7016c5d\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " Oct 29 23:32:45.229132 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.229110 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-credential-socket\") pod \"77d46565-e43f-44a8-8d78-4417e7016c5d\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " Oct 29 23:32:45.229601 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.229195 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77d46565-e43f-44a8-8d78-4417e7016c5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d46565-e43f-44a8-8d78-4417e7016c5d-istiod-ca-cert\") pod \"77d46565-e43f-44a8-8d78-4417e7016c5d\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " Oct 29 23:32:45.229601 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.229257 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d46565-e43f-44a8-8d78-4417e7016c5d-kube-api-access-s9mhm\" (UniqueName: \"kubernetes.io/projected/77d46565-e43f-44a8-8d78-4417e7016c5d-kube-api-access-s9mhm\") pod \"77d46565-e43f-44a8-8d78-4417e7016c5d\" (UID: \"77d46565-e43f-44a8-8d78-4417e7016c5d\") " Oct 29 23:32:45.229601 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.229454 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-data" pod "77d46565-e43f-44a8-8d78-4417e7016c5d" (UID: "77d46565-e43f-44a8-8d78-4417e7016c5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:45.229601 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.229455 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-workload-certs" pod "77d46565-e43f-44a8-8d78-4417e7016c5d" (UID: "77d46565-e43f-44a8-8d78-4417e7016c5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:45.229601 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.229476 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-workload-socket" pod "77d46565-e43f-44a8-8d78-4417e7016c5d" (UID: "77d46565-e43f-44a8-8d78-4417e7016c5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:45.229904 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.229603 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-credential-socket" pod "77d46565-e43f-44a8-8d78-4417e7016c5d" (UID: "77d46565-e43f-44a8-8d78-4417e7016c5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:45.229904 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.229735 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d46565-e43f-44a8-8d78-4417e7016c5d-istiod-ca-cert" pod "77d46565-e43f-44a8-8d78-4417e7016c5d" (UID: "77d46565-e43f-44a8-8d78-4417e7016c5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:45.229904 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.229799 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:45.229904 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.229819 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:45.229904 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.229833 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:45.229904 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.229846 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:45.231483 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.231459 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-podinfo" pod "77d46565-e43f-44a8-8d78-4417e7016c5d" (UID: "77d46565-e43f-44a8-8d78-4417e7016c5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:32:45.231614 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.231554 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-token" pod "77d46565-e43f-44a8-8d78-4417e7016c5d" (UID: "77d46565-e43f-44a8-8d78-4417e7016c5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:45.232158 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.232130 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d46565-e43f-44a8-8d78-4417e7016c5d-kube-api-access-s9mhm" pod "77d46565-e43f-44a8-8d78-4417e7016c5d" (UID: "77d46565-e43f-44a8-8d78-4417e7016c5d"). InnerVolumeSpecName "kube-api-access-s9mhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:45.232298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.232223 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-envoy" pod "77d46565-e43f-44a8-8d78-4417e7016c5d" (UID: "77d46565-e43f-44a8-8d78-4417e7016c5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:45.331139 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.331099 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:45.331139 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.331133 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:45.331139 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.331142 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d46565-e43f-44a8-8d78-4417e7016c5d-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:45.331139 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.331149 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d46565-e43f-44a8-8d78-4417e7016c5d-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:45.331470 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.331158 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9mhm\" (UniqueName: \"kubernetes.io/projected/77d46565-e43f-44a8-8d78-4417e7016c5d-kube-api-access-s9mhm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:45.561645 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.561612 4821 generic.go:356] "Generic (PLEG): container finished" podID="77d46565-e43f-44a8-8d78-4417e7016c5d" containerID="2147d171319ef1ed54a31f87fa4113cc09f04fdff294d37bee328d4640bf3e07" exitCode=0 Oct 29 23:32:45.561825 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.561655 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" Oct 29 23:32:45.561825 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.561693 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" event={"ID":"77d46565-e43f-44a8-8d78-4417e7016c5d","Type":"ContainerDied","Data":"2147d171319ef1ed54a31f87fa4113cc09f04fdff294d37bee328d4640bf3e07"} Oct 29 23:32:45.561825 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.561749 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw" event={"ID":"77d46565-e43f-44a8-8d78-4417e7016c5d","Type":"ContainerDied","Data":"abd652882197937d72079ab98ac6fe6da75ad2fce09525ba205e8d46f5ad10c9"} Oct 29 23:32:45.561825 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.561771 4821 scope.go:122] "RemoveContainer" containerID="2147d171319ef1ed54a31f87fa4113cc09f04fdff294d37bee328d4640bf3e07" Oct 29 23:32:45.571949 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.571932 4821 scope.go:122] "RemoveContainer" containerID="2147d171319ef1ed54a31f87fa4113cc09f04fdff294d37bee328d4640bf3e07" Oct 29 23:32:45.572241 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:32:45.572194 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2147d171319ef1ed54a31f87fa4113cc09f04fdff294d37bee328d4640bf3e07\": container with ID starting with 2147d171319ef1ed54a31f87fa4113cc09f04fdff294d37bee328d4640bf3e07 not found: ID does not exist" containerID="2147d171319ef1ed54a31f87fa4113cc09f04fdff294d37bee328d4640bf3e07" Oct 29 23:32:45.572360 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.572246 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2147d171319ef1ed54a31f87fa4113cc09f04fdff294d37bee328d4640bf3e07"} err="failed to get container status \"2147d171319ef1ed54a31f87fa4113cc09f04fdff294d37bee328d4640bf3e07\": rpc error: code = NotFound desc = could not find container \"2147d171319ef1ed54a31f87fa4113cc09f04fdff294d37bee328d4640bf3e07\": container with ID starting with 2147d171319ef1ed54a31f87fa4113cc09f04fdff294d37bee328d4640bf3e07 not found: ID does not exist" Oct 29 23:32:45.617350 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.617317 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw"] Oct 29 23:32:45.653724 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:45.653680 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af237d-openshift-default-64f945b797-jjtmw"] Oct 29 23:32:46.170637 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:46.170606 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77d46565-e43f-44a8-8d78-4417e7016c5d" path="/var/lib/kubelet/pods/77d46565-e43f-44a8-8d78-4417e7016c5d/volumes" Oct 29 23:32:51.749884 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.749846 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-482f7889-lk7kd"] Oct 29 23:32:51.750475 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.750454 4821 cpu_manager.go:414] "containerMap: removing container" podUID="36ea1460-ac2f-410a-b06a-b18f1f8e56d3" containerName="istio-proxy" Oct 29 23:32:51.750520 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.750480 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ea1460-ac2f-410a-b06a-b18f1f8e56d3" containerName="istio-proxy" Oct 29 23:32:51.750520 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.750499 4821 cpu_manager.go:414] "containerMap: removing container" podUID="77d46565-e43f-44a8-8d78-4417e7016c5d" containerName="istio-proxy" Oct 29 23:32:51.750520 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.750507 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77d46565-e43f-44a8-8d78-4417e7016c5d" containerName="istio-proxy" Oct 29 23:32:51.750620 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.750528 4821 cpu_manager.go:414] "containerMap: removing container" podUID="36ea1460-ac2f-410a-b06a-b18f1f8e56d3" containerName="istio-proxy" Oct 29 23:32:51.750620 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.750536 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ea1460-ac2f-410a-b06a-b18f1f8e56d3" containerName="istio-proxy" Oct 29 23:32:51.750620 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.750561 4821 cpu_manager.go:414] "containerMap: removing container" podUID="7ed236b2-7d1a-46a6-bbb9-55683420c3ed" containerName="cleanup" Oct 29 23:32:51.750620 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.750568 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed236b2-7d1a-46a6-bbb9-55683420c3ed" containerName="cleanup" Oct 29 23:32:51.750620 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.750595 4821 cpu_manager.go:414] "containerMap: removing container" podUID="cafaae7c-1ed2-40de-949a-418034ce834f" containerName="revoke-keys" Oct 29 23:32:51.750620 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.750601 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cafaae7c-1ed2-40de-949a-418034ce834f" containerName="revoke-keys" Oct 29 23:32:51.750620 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.750617 4821 cpu_manager.go:414] "containerMap: removing container" podUID="bc852c3e-5277-4f30-9063-f4901fdca67e" containerName="cleanup" Oct 29 23:32:51.750829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.750625 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc852c3e-5277-4f30-9063-f4901fdca67e" containerName="cleanup" Oct 29 23:32:51.750829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.750633 4821 cpu_manager.go:414] "containerMap: removing container" podUID="65b14de3-46e0-438e-9e72-68ab3c19f4b3" containerName="payload-processing" Oct 29 23:32:51.750829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.750639 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b14de3-46e0-438e-9e72-68ab3c19f4b3" containerName="payload-processing" Oct 29 23:32:51.750829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.750749 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed236b2-7d1a-46a6-bbb9-55683420c3ed" containerName="cleanup" Oct 29 23:32:51.750829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.750763 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="cafaae7c-1ed2-40de-949a-418034ce834f" containerName="revoke-keys" Oct 29 23:32:51.750829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.750776 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b14de3-46e0-438e-9e72-68ab3c19f4b3" containerName="payload-processing" Oct 29 23:32:51.750829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.750786 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ea1460-ac2f-410a-b06a-b18f1f8e56d3" containerName="istio-proxy" Oct 29 23:32:51.750829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.750794 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="77d46565-e43f-44a8-8d78-4417e7016c5d" containerName="istio-proxy" Oct 29 23:32:51.750829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.750804 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ea1460-ac2f-410a-b06a-b18f1f8e56d3" containerName="istio-proxy" Oct 29 23:32:51.750829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.750812 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc852c3e-5277-4f30-9063-f4901fdca67e" containerName="cleanup" Oct 29 23:32:51.750829 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.750819 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="99fcc999-240e-4254-9926-d1af30f9f8f8" containerName="cleanup" Oct 29 23:32:51.756119 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.756094 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-482f7889-lk7kd" Oct 29 23:32:51.774262 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.774234 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-482f7889-lk7kd"] Oct 29 23:32:51.835475 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.835417 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qcgk\" (UniqueName: \"kubernetes.io/projected/85996d66-20b5-4d3f-b915-8172999544ae-kube-api-access-4qcgk\") pod \"maas-api-revoke-sub-482f7889-lk7kd\" (UID: \"85996d66-20b5-4d3f-b915-8172999544ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482f7889-lk7kd" Oct 29 23:32:51.835667 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.835579 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85996d66-20b5-4d3f-b915-8172999544ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-482f7889-lk7kd\" (UID: \"85996d66-20b5-4d3f-b915-8172999544ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482f7889-lk7kd" Oct 29 23:32:51.835762 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.835702 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85996d66-20b5-4d3f-b915-8172999544ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-482f7889-lk7kd\" (UID: \"85996d66-20b5-4d3f-b915-8172999544ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482f7889-lk7kd" Oct 29 23:32:51.839124 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.839101 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:32:51.839124 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.839126 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dmx57\"" Oct 29 23:32:51.937138 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.937109 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85996d66-20b5-4d3f-b915-8172999544ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-482f7889-lk7kd\" (UID: \"85996d66-20b5-4d3f-b915-8172999544ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482f7889-lk7kd" Oct 29 23:32:51.937355 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.937158 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85996d66-20b5-4d3f-b915-8172999544ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-482f7889-lk7kd\" (UID: \"85996d66-20b5-4d3f-b915-8172999544ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482f7889-lk7kd" Oct 29 23:32:51.937355 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.937238 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qcgk\" (UniqueName: \"kubernetes.io/projected/85996d66-20b5-4d3f-b915-8172999544ae-kube-api-access-4qcgk\") pod \"maas-api-revoke-sub-482f7889-lk7kd\" (UID: \"85996d66-20b5-4d3f-b915-8172999544ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482f7889-lk7kd" Oct 29 23:32:51.937899 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.937870 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85996d66-20b5-4d3f-b915-8172999544ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-482f7889-lk7kd\" (UID: \"85996d66-20b5-4d3f-b915-8172999544ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482f7889-lk7kd" Oct 29 23:32:51.939658 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.939635 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85996d66-20b5-4d3f-b915-8172999544ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-482f7889-lk7kd\" (UID: \"85996d66-20b5-4d3f-b915-8172999544ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482f7889-lk7kd" Oct 29 23:32:51.964502 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:51.964480 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qcgk\" (UniqueName: \"kubernetes.io/projected/85996d66-20b5-4d3f-b915-8172999544ae-kube-api-access-4qcgk\") pod \"maas-api-revoke-sub-482f7889-lk7kd\" (UID: \"85996d66-20b5-4d3f-b915-8172999544ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482f7889-lk7kd" Oct 29 23:32:52.073597 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:52.073518 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-482f7889-lk7kd" Oct 29 23:32:52.378691 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:52.378661 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-482f7889-lk7kd"] Oct 29 23:32:52.380956 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:32:52.380922 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85996d66_20b5_4d3f_b915_8172999544ae.slice/crio-7180462dd3f91946f34cc70e91f4f586cf338c1489ef60135e30c3aac18bbc52 WatchSource:0}: Error finding container 7180462dd3f91946f34cc70e91f4f586cf338c1489ef60135e30c3aac18bbc52: Status 404 returned error can't find the container with id 7180462dd3f91946f34cc70e91f4f586cf338c1489ef60135e30c3aac18bbc52 Oct 29 23:32:52.604679 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:52.604645 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-482f7889-lk7kd" event={"ID":"85996d66-20b5-4d3f-b915-8172999544ae","Type":"ContainerStarted","Data":"7180462dd3f91946f34cc70e91f4f586cf338c1489ef60135e30c3aac18bbc52"} Oct 29 23:32:53.614219 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:53.614176 4821 generic.go:356] "Generic (PLEG): container finished" podID="85996d66-20b5-4d3f-b915-8172999544ae" containerID="ebfa965e4b2e55e4daffef9929acafc92f36f045631db2fcad2e847d11b646c0" exitCode=0 Oct 29 23:32:53.614647 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:53.614237 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-482f7889-lk7kd" event={"ID":"85996d66-20b5-4d3f-b915-8172999544ae","Type":"ContainerDied","Data":"ebfa965e4b2e55e4daffef9929acafc92f36f045631db2fcad2e847d11b646c0"} Oct 29 23:32:53.884770 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:32:53.884672 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:32:54.131672 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:54.131632 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf"] Oct 29 23:32:54.133974 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:54.133947 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4504f1-8cwxf"] Oct 29 23:32:54.174946 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:54.174870 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cafaae7c-1ed2-40de-949a-418034ce834f" path="/var/lib/kubelet/pods/cafaae7c-1ed2-40de-949a-418034ce834f/volumes" Oct 29 23:32:54.717099 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:54.717047 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" containerID="cri-o://0f2658ce58eb7f018465f05bf9df5593e4b69dc85fd69d3ab6003ffcaf2aab04" gracePeriod=45 Oct 29 23:32:54.777626 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:54.777570 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" containerID="cri-o://abbdb33664418d4d2f2d2b3ffc4859dda93e43aa0d1dbcddaf78c0818dcd9fef" gracePeriod=45 Oct 29 23:32:54.891149 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:54.891125 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-482f7889-lk7kd" Oct 29 23:32:54.977011 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:54.976982 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85996d66-20b5-4d3f-b915-8172999544ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85996d66-20b5-4d3f-b915-8172999544ae-maas-api-xks-ca\") pod \"85996d66-20b5-4d3f-b915-8172999544ae\" (UID: \"85996d66-20b5-4d3f-b915-8172999544ae\") " Oct 29 23:32:54.977175 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:54.977042 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85996d66-20b5-4d3f-b915-8172999544ae-kube-api-access-4qcgk\" (UniqueName: \"kubernetes.io/projected/85996d66-20b5-4d3f-b915-8172999544ae-kube-api-access-4qcgk\") pod \"85996d66-20b5-4d3f-b915-8172999544ae\" (UID: \"85996d66-20b5-4d3f-b915-8172999544ae\") " Oct 29 23:32:54.977175 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:54.977080 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85996d66-20b5-4d3f-b915-8172999544ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85996d66-20b5-4d3f-b915-8172999544ae-maas-api-openshift-ca\") pod \"85996d66-20b5-4d3f-b915-8172999544ae\" (UID: \"85996d66-20b5-4d3f-b915-8172999544ae\") " Oct 29 23:32:54.977461 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:54.977433 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85996d66-20b5-4d3f-b915-8172999544ae-maas-api-openshift-ca" pod "85996d66-20b5-4d3f-b915-8172999544ae" (UID: "85996d66-20b5-4d3f-b915-8172999544ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:32:54.977551 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:54.977528 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85996d66-20b5-4d3f-b915-8172999544ae-maas-api-openshift-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:54.979737 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:54.979700 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85996d66-20b5-4d3f-b915-8172999544ae-maas-api-xks-ca" pod "85996d66-20b5-4d3f-b915-8172999544ae" (UID: "85996d66-20b5-4d3f-b915-8172999544ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:32:54.979864 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:54.979841 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85996d66-20b5-4d3f-b915-8172999544ae-kube-api-access-4qcgk" pod "85996d66-20b5-4d3f-b915-8172999544ae" (UID: "85996d66-20b5-4d3f-b915-8172999544ae"). InnerVolumeSpecName "kube-api-access-4qcgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:55.049288 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.044963 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 23:32:55.078335 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.078305 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85996d66-20b5-4d3f-b915-8172999544ae-maas-api-xks-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:55.078335 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.078337 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qcgk\" (UniqueName: \"kubernetes.io/projected/85996d66-20b5-4d3f-b915-8172999544ae-kube-api-access-4qcgk\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:55.118070 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.118039 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 23:32:55.179282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.179188 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-tmp-dir\") pod \"1d3f4d01-15ca-4d75-a69b-a5adff178017\" (UID: \"1d3f4d01-15ca-4d75-a69b-a5adff178017\") " Oct 29 23:32:55.179282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.179264 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d3f4d01-15ca-4d75-a69b-a5adff178017-kube-api-access-6vqjv\" (UniqueName: \"kubernetes.io/projected/1d3f4d01-15ca-4d75-a69b-a5adff178017-kube-api-access-6vqjv\") pod \"1d3f4d01-15ca-4d75-a69b-a5adff178017\" (UID: \"1d3f4d01-15ca-4d75-a69b-a5adff178017\") " Oct 29 23:32:55.179564 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.179305 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-home\" (UniqueName: \"kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-home\") pod \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\" (UID: \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\") " Oct 29 23:32:55.179564 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.179346 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-model-cache\") pod \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\" (UID: \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\") " Oct 29 23:32:55.179564 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.179388 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-dshm\") pod \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\" (UID: \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\") " Oct 29 23:32:55.179564 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.179455 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-tmp-dir" pod "1d3f4d01-15ca-4d75-a69b-a5adff178017" (UID: "1d3f4d01-15ca-4d75-a69b-a5adff178017"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:55.179757 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.179576 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-kube-api-access-mqrhk\" (UniqueName: \"kubernetes.io/projected/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-kube-api-access-mqrhk\") pod \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\" (UID: \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\") " Oct 29 23:32:55.179757 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.179643 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-home" pod "59c07a4a-dee9-4e00-a10b-eee2f9160fd1" (UID: "59c07a4a-dee9-4e00-a10b-eee2f9160fd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:55.179757 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.179719 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-model-cache" pod "59c07a4a-dee9-4e00-a10b-eee2f9160fd1" (UID: "59c07a4a-dee9-4e00-a10b-eee2f9160fd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:55.179969 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.179639 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d3f4d01-15ca-4d75-a69b-a5adff178017-tls-certs\" (UniqueName: \"kubernetes.io/secret/1d3f4d01-15ca-4d75-a69b-a5adff178017-tls-certs\") pod \"1d3f4d01-15ca-4d75-a69b-a5adff178017\" (UID: \"1d3f4d01-15ca-4d75-a69b-a5adff178017\") " Oct 29 23:32:55.180114 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.180013 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-home\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-home\") pod \"1d3f4d01-15ca-4d75-a69b-a5adff178017\" (UID: \"1d3f4d01-15ca-4d75-a69b-a5adff178017\") " Oct 29 23:32:55.180114 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.180042 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-dshm\") pod \"1d3f4d01-15ca-4d75-a69b-a5adff178017\" (UID: \"1d3f4d01-15ca-4d75-a69b-a5adff178017\") " Oct 29 23:32:55.180114 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.180110 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-tls-certs\" (UniqueName: \"kubernetes.io/secret/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-tls-certs\") pod \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\" (UID: \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\") " Oct 29 23:32:55.181605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.180193 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-model-cache\") pod \"1d3f4d01-15ca-4d75-a69b-a5adff178017\" (UID: \"1d3f4d01-15ca-4d75-a69b-a5adff178017\") " Oct 29 23:32:55.181605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.180240 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-tmp-dir\") pod \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\" (UID: \"59c07a4a-dee9-4e00-a10b-eee2f9160fd1\") " Oct 29 23:32:55.181605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.180885 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:55.181605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.180906 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:55.181605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.180920 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:55.181605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.181188 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-tmp-dir" pod "59c07a4a-dee9-4e00-a10b-eee2f9160fd1" (UID: "59c07a4a-dee9-4e00-a10b-eee2f9160fd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:55.181605 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.181455 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-model-cache" pod "1d3f4d01-15ca-4d75-a69b-a5adff178017" (UID: "1d3f4d01-15ca-4d75-a69b-a5adff178017"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:55.182020 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.181629 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-home" pod "1d3f4d01-15ca-4d75-a69b-a5adff178017" (UID: "1d3f4d01-15ca-4d75-a69b-a5adff178017"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:55.186330 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.186307 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-dshm" pod "1d3f4d01-15ca-4d75-a69b-a5adff178017" (UID: "1d3f4d01-15ca-4d75-a69b-a5adff178017"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:55.186431 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.186337 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d3f4d01-15ca-4d75-a69b-a5adff178017-tls-certs" pod "1d3f4d01-15ca-4d75-a69b-a5adff178017" (UID: "1d3f4d01-15ca-4d75-a69b-a5adff178017"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:32:55.186431 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.186336 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-dshm" pod "59c07a4a-dee9-4e00-a10b-eee2f9160fd1" (UID: "59c07a4a-dee9-4e00-a10b-eee2f9160fd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:32:55.186431 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.186356 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-kube-api-access-mqrhk" pod "59c07a4a-dee9-4e00-a10b-eee2f9160fd1" (UID: "59c07a4a-dee9-4e00-a10b-eee2f9160fd1"). InnerVolumeSpecName "kube-api-access-mqrhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:55.186431 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.186337 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d3f4d01-15ca-4d75-a69b-a5adff178017-kube-api-access-6vqjv" pod "1d3f4d01-15ca-4d75-a69b-a5adff178017" (UID: "1d3f4d01-15ca-4d75-a69b-a5adff178017"). InnerVolumeSpecName "kube-api-access-6vqjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:32:55.186431 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.186350 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-tls-certs" pod "59c07a4a-dee9-4e00-a10b-eee2f9160fd1" (UID: "59c07a4a-dee9-4e00-a10b-eee2f9160fd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:32:55.281903 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.281870 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:55.281903 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.281898 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqrhk\" (UniqueName: \"kubernetes.io/projected/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-kube-api-access-mqrhk\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:55.281903 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.281909 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d3f4d01-15ca-4d75-a69b-a5adff178017-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:55.282138 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.281918 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:55.282138 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.281926 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:55.282138 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.281933 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:55.282138 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.281940 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d3f4d01-15ca-4d75-a69b-a5adff178017-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:55.282138 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.281947 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59c07a4a-dee9-4e00-a10b-eee2f9160fd1-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:55.282138 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.281954 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vqjv\" (UniqueName: \"kubernetes.io/projected/1d3f4d01-15ca-4d75-a69b-a5adff178017-kube-api-access-6vqjv\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:32:55.627313 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.627283 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-482f7889-lk7kd" Oct 29 23:32:55.627591 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.627311 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-482f7889-lk7kd" event={"ID":"85996d66-20b5-4d3f-b915-8172999544ae","Type":"ContainerDied","Data":"7180462dd3f91946f34cc70e91f4f586cf338c1489ef60135e30c3aac18bbc52"} Oct 29 23:32:55.627591 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.627360 4821 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7180462dd3f91946f34cc70e91f4f586cf338c1489ef60135e30c3aac18bbc52" Oct 29 23:32:55.628849 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.628823 4821 generic.go:356] "Generic (PLEG): container finished" podID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerID="abbdb33664418d4d2f2d2b3ffc4859dda93e43aa0d1dbcddaf78c0818dcd9fef" exitCode=0 Oct 29 23:32:55.628969 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.628852 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" event={"ID":"1d3f4d01-15ca-4d75-a69b-a5adff178017","Type":"ContainerDied","Data":"abbdb33664418d4d2f2d2b3ffc4859dda93e43aa0d1dbcddaf78c0818dcd9fef"} Oct 29 23:32:55.628969 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.628882 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" Oct 29 23:32:55.628969 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.628899 4821 scope.go:122] "RemoveContainer" containerID="abbdb33664418d4d2f2d2b3ffc4859dda93e43aa0d1dbcddaf78c0818dcd9fef" Oct 29 23:32:55.629116 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.628888 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" event={"ID":"1d3f4d01-15ca-4d75-a69b-a5adff178017","Type":"ContainerDied","Data":"b1e726d9dcbb2e5e54a70666cbf073d5fc16499a33aeac75862bc5b8e6d22a98"} Oct 29 23:32:55.630445 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.630422 4821 generic.go:356] "Generic (PLEG): container finished" podID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerID="0f2658ce58eb7f018465f05bf9df5593e4b69dc85fd69d3ab6003ffcaf2aab04" exitCode=0 Oct 29 23:32:55.630501 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.630456 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" Oct 29 23:32:55.630501 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.630455 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" event={"ID":"59c07a4a-dee9-4e00-a10b-eee2f9160fd1","Type":"ContainerDied","Data":"0f2658ce58eb7f018465f05bf9df5593e4b69dc85fd69d3ab6003ffcaf2aab04"} Oct 29 23:32:55.630501 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.630487 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v" event={"ID":"59c07a4a-dee9-4e00-a10b-eee2f9160fd1","Type":"ContainerDied","Data":"7ae1996dd5e83892b308ff7cca7c6137f03742ac0e55a784422f1785b804d191"} Oct 29 23:32:55.640048 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.640029 4821 scope.go:122] "RemoveContainer" containerID="abbdb33664418d4d2f2d2b3ffc4859dda93e43aa0d1dbcddaf78c0818dcd9fef" Oct 29 23:32:55.640390 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:32:55.640359 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abbdb33664418d4d2f2d2b3ffc4859dda93e43aa0d1dbcddaf78c0818dcd9fef\": container with ID starting with abbdb33664418d4d2f2d2b3ffc4859dda93e43aa0d1dbcddaf78c0818dcd9fef not found: ID does not exist" containerID="abbdb33664418d4d2f2d2b3ffc4859dda93e43aa0d1dbcddaf78c0818dcd9fef" Oct 29 23:32:55.640474 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.640396 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abbdb33664418d4d2f2d2b3ffc4859dda93e43aa0d1dbcddaf78c0818dcd9fef"} err="failed to get container status \"abbdb33664418d4d2f2d2b3ffc4859dda93e43aa0d1dbcddaf78c0818dcd9fef\": rpc error: code = NotFound desc = could not find container \"abbdb33664418d4d2f2d2b3ffc4859dda93e43aa0d1dbcddaf78c0818dcd9fef\": container with ID starting with abbdb33664418d4d2f2d2b3ffc4859dda93e43aa0d1dbcddaf78c0818dcd9fef not found: ID does not exist" Oct 29 23:32:55.640474 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.640424 4821 scope.go:122] "RemoveContainer" containerID="0f2658ce58eb7f018465f05bf9df5593e4b69dc85fd69d3ab6003ffcaf2aab04" Oct 29 23:32:55.678079 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.678054 4821 scope.go:122] "RemoveContainer" containerID="0f2658ce58eb7f018465f05bf9df5593e4b69dc85fd69d3ab6003ffcaf2aab04" Oct 29 23:32:55.678397 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:32:55.678372 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f2658ce58eb7f018465f05bf9df5593e4b69dc85fd69d3ab6003ffcaf2aab04\": container with ID starting with 0f2658ce58eb7f018465f05bf9df5593e4b69dc85fd69d3ab6003ffcaf2aab04 not found: ID does not exist" containerID="0f2658ce58eb7f018465f05bf9df5593e4b69dc85fd69d3ab6003ffcaf2aab04" Oct 29 23:32:55.678469 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.678406 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f2658ce58eb7f018465f05bf9df5593e4b69dc85fd69d3ab6003ffcaf2aab04"} err="failed to get container status \"0f2658ce58eb7f018465f05bf9df5593e4b69dc85fd69d3ab6003ffcaf2aab04\": rpc error: code = NotFound desc = could not find container \"0f2658ce58eb7f018465f05bf9df5593e4b69dc85fd69d3ab6003ffcaf2aab04\": container with ID starting with 0f2658ce58eb7f018465f05bf9df5593e4b69dc85fd69d3ab6003ffcaf2aab04 not found: ID does not exist" Oct 29 23:32:55.702809 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.702782 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w"] Oct 29 23:32:55.716048 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.716020 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w"] Oct 29 23:32:55.745540 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.745516 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v"] Oct 29 23:32:55.749421 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.749397 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e131af/distinct-2-w1-e131af-kserve-7486ff57db-bbv9v"] Oct 29 23:32:55.991569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:55.991482 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e131af/simulator-w1-e131af-kserve-6c64989687-qpx6w" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 29 23:32:56.169347 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:56.169317 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" path="/var/lib/kubelet/pods/1d3f4d01-15ca-4d75-a69b-a5adff178017/volumes" Oct 29 23:32:56.169740 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:32:56.169728 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" path="/var/lib/kubelet/pods/59c07a4a-dee9-4e00-a10b-eee2f9160fd1/volumes" Oct 29 23:33:04.853176 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:04.853140 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2"] Oct 29 23:33:04.853947 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:04.853915 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" podUID="f3f2d74f-e211-41ce-8ed6-432892f64365" containerName="istio-proxy" containerID="cri-o://3b9f57fe212d1a4ddc05770d5f93e3870c8767479b97db698371e6cd9413b369" gracePeriod=30 Oct 29 23:33:10.112267 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.112235 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:33:10.176595 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.176500 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f2d74f-e211-41ce-8ed6-432892f64365-kube-api-access-7msqw\" (UniqueName: \"kubernetes.io/projected/f3f2d74f-e211-41ce-8ed6-432892f64365-kube-api-access-7msqw\") pod \"f3f2d74f-e211-41ce-8ed6-432892f64365\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " Oct 29 23:33:10.176595 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.176557 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-token\" (UniqueName: \"kubernetes.io/projected/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-token\") pod \"f3f2d74f-e211-41ce-8ed6-432892f64365\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " Oct 29 23:33:10.176823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.176605 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-workload-certs\") pod \"f3f2d74f-e211-41ce-8ed6-432892f64365\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " Oct 29 23:33:10.176823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.176639 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-credential-socket\") pod \"f3f2d74f-e211-41ce-8ed6-432892f64365\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " Oct 29 23:33:10.176823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.176674 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-data\") pod \"f3f2d74f-e211-41ce-8ed6-432892f64365\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " Oct 29 23:33:10.176823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.176714 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-workload-socket\") pod \"f3f2d74f-e211-41ce-8ed6-432892f64365\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " Oct 29 23:33:10.176823 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.176785 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f2d74f-e211-41ce-8ed6-432892f64365-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f2d74f-e211-41ce-8ed6-432892f64365-istiod-ca-cert\") pod \"f3f2d74f-e211-41ce-8ed6-432892f64365\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " Oct 29 23:33:10.177083 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.176849 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-podinfo\") pod \"f3f2d74f-e211-41ce-8ed6-432892f64365\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " Oct 29 23:33:10.177083 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.176918 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-envoy\") pod \"f3f2d74f-e211-41ce-8ed6-432892f64365\" (UID: \"f3f2d74f-e211-41ce-8ed6-432892f64365\") " Oct 29 23:33:10.177083 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.176936 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-workload-certs" pod "f3f2d74f-e211-41ce-8ed6-432892f64365" (UID: "f3f2d74f-e211-41ce-8ed6-432892f64365"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:10.177316 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.177273 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-data" pod "f3f2d74f-e211-41ce-8ed6-432892f64365" (UID: "f3f2d74f-e211-41ce-8ed6-432892f64365"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:10.177465 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.177348 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-credential-socket" pod "f3f2d74f-e211-41ce-8ed6-432892f64365" (UID: "f3f2d74f-e211-41ce-8ed6-432892f64365"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:10.177465 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.177394 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:10.177465 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.177438 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:10.177722 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.177507 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-workload-socket" pod "f3f2d74f-e211-41ce-8ed6-432892f64365" (UID: "f3f2d74f-e211-41ce-8ed6-432892f64365"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:10.177930 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.177839 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f2d74f-e211-41ce-8ed6-432892f64365-istiod-ca-cert" pod "f3f2d74f-e211-41ce-8ed6-432892f64365" (UID: "f3f2d74f-e211-41ce-8ed6-432892f64365"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:10.179376 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.179350 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-token" pod "f3f2d74f-e211-41ce-8ed6-432892f64365" (UID: "f3f2d74f-e211-41ce-8ed6-432892f64365"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:10.179547 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.179529 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-envoy" pod "f3f2d74f-e211-41ce-8ed6-432892f64365" (UID: "f3f2d74f-e211-41ce-8ed6-432892f64365"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:10.179658 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.179639 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f2d74f-e211-41ce-8ed6-432892f64365-kube-api-access-7msqw" pod "f3f2d74f-e211-41ce-8ed6-432892f64365" (UID: "f3f2d74f-e211-41ce-8ed6-432892f64365"). InnerVolumeSpecName "kube-api-access-7msqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:10.179896 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.179873 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-podinfo" pod "f3f2d74f-e211-41ce-8ed6-432892f64365" (UID: "f3f2d74f-e211-41ce-8ed6-432892f64365"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:33:10.278465 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.278428 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7msqw\" (UniqueName: \"kubernetes.io/projected/f3f2d74f-e211-41ce-8ed6-432892f64365-kube-api-access-7msqw\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:10.278465 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.278458 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:10.278465 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.278472 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:10.278712 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.278484 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:10.278712 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.278498 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f2d74f-e211-41ce-8ed6-432892f64365-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:10.278712 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.278509 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:10.278712 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.278519 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f2d74f-e211-41ce-8ed6-432892f64365-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:10.338070 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.338035 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm"] Oct 29 23:33:10.353513 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.353478 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx"] Oct 29 23:33:10.726662 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.726624 4821 generic.go:356] "Generic (PLEG): container finished" podID="f3f2d74f-e211-41ce-8ed6-432892f64365" containerID="3b9f57fe212d1a4ddc05770d5f93e3870c8767479b97db698371e6cd9413b369" exitCode=0 Oct 29 23:33:10.726872 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.726696 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" Oct 29 23:33:10.726872 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.726721 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" event={"ID":"f3f2d74f-e211-41ce-8ed6-432892f64365","Type":"ContainerDied","Data":"3b9f57fe212d1a4ddc05770d5f93e3870c8767479b97db698371e6cd9413b369"} Oct 29 23:33:10.726872 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.726794 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2" event={"ID":"f3f2d74f-e211-41ce-8ed6-432892f64365","Type":"ContainerDied","Data":"a55aca95be1da619929763b72991c43cc2379d4b8d4e85f3edc680da52d58d4e"} Oct 29 23:33:10.726872 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.726829 4821 scope.go:122] "RemoveContainer" containerID="3b9f57fe212d1a4ddc05770d5f93e3870c8767479b97db698371e6cd9413b369" Oct 29 23:33:10.740882 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.740857 4821 scope.go:122] "RemoveContainer" containerID="646da0945b2d2ef89e14a88a15f2757d0928bb9e4ad2381bf453824fa47e4184" Oct 29 23:33:10.752276 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.752262 4821 scope.go:122] "RemoveContainer" containerID="3b9f57fe212d1a4ddc05770d5f93e3870c8767479b97db698371e6cd9413b369" Oct 29 23:33:10.752572 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:33:10.752542 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b9f57fe212d1a4ddc05770d5f93e3870c8767479b97db698371e6cd9413b369\": container with ID starting with 3b9f57fe212d1a4ddc05770d5f93e3870c8767479b97db698371e6cd9413b369 not found: ID does not exist" containerID="3b9f57fe212d1a4ddc05770d5f93e3870c8767479b97db698371e6cd9413b369" Oct 29 23:33:10.752638 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.752583 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9f57fe212d1a4ddc05770d5f93e3870c8767479b97db698371e6cd9413b369"} err="failed to get container status \"3b9f57fe212d1a4ddc05770d5f93e3870c8767479b97db698371e6cd9413b369\": rpc error: code = NotFound desc = could not find container \"3b9f57fe212d1a4ddc05770d5f93e3870c8767479b97db698371e6cd9413b369\": container with ID starting with 3b9f57fe212d1a4ddc05770d5f93e3870c8767479b97db698371e6cd9413b369 not found: ID does not exist" Oct 29 23:33:10.752638 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.752600 4821 scope.go:122] "RemoveContainer" containerID="646da0945b2d2ef89e14a88a15f2757d0928bb9e4ad2381bf453824fa47e4184" Oct 29 23:33:10.752863 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:33:10.752838 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"646da0945b2d2ef89e14a88a15f2757d0928bb9e4ad2381bf453824fa47e4184\": container with ID starting with 646da0945b2d2ef89e14a88a15f2757d0928bb9e4ad2381bf453824fa47e4184 not found: ID does not exist" containerID="646da0945b2d2ef89e14a88a15f2757d0928bb9e4ad2381bf453824fa47e4184" Oct 29 23:33:10.752915 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.752873 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646da0945b2d2ef89e14a88a15f2757d0928bb9e4ad2381bf453824fa47e4184"} err="failed to get container status \"646da0945b2d2ef89e14a88a15f2757d0928bb9e4ad2381bf453824fa47e4184\": rpc error: code = NotFound desc = could not find container \"646da0945b2d2ef89e14a88a15f2757d0928bb9e4ad2381bf453824fa47e4184\": container with ID starting with 646da0945b2d2ef89e14a88a15f2757d0928bb9e4ad2381bf453824fa47e4184 not found: ID does not exist" Oct 29 23:33:10.785216 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.785168 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2"] Oct 29 23:33:10.789072 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:10.789048 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-65e2d3-openshift-default-5c656cf67f-s5hm2"] Oct 29 23:33:12.168690 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:12.168658 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f2d74f-e211-41ce-8ed6-432892f64365" path="/var/lib/kubelet/pods/f3f2d74f-e211-41ce-8ed6-432892f64365/volumes" Oct 29 23:33:14.729677 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:14.729640 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4"] Oct 29 23:33:14.732541 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:14.732504 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" podUID="9be68e75-df62-4a43-9ae9-20c04683f4c7" containerName="istio-proxy" containerID="cri-o://7934b7a54f73d2c8220c99250e75365b6e5c194a793de9b3ca9125e554500004" gracePeriod=30 Oct 29 23:33:16.176751 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:16.176717 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz"] Oct 29 23:33:16.208020 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:16.207992 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt"] Oct 29 23:33:17.089265 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:17.089229 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp"] Oct 29 23:33:17.089534 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:17.089479 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp" podUID="bab23b35-8838-415c-90bf-aebf41448b58" containerName="maas-api" containerID="cri-o://b33053c0bc1933c80d4e767a974d2bd8dda3e82ca6e73778c81f61c5cd218838" gracePeriod=30 Oct 29 23:33:17.359798 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:17.359773 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp" Oct 29 23:33:17.449535 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:17.449502 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bab23b35-8838-415c-90bf-aebf41448b58-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bab23b35-8838-415c-90bf-aebf41448b58-maas-api-tls\") pod \"bab23b35-8838-415c-90bf-aebf41448b58\" (UID: \"bab23b35-8838-415c-90bf-aebf41448b58\") " Oct 29 23:33:17.449716 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:17.449544 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bab23b35-8838-415c-90bf-aebf41448b58-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bab23b35-8838-415c-90bf-aebf41448b58-metrics-certs\") pod \"bab23b35-8838-415c-90bf-aebf41448b58\" (UID: \"bab23b35-8838-415c-90bf-aebf41448b58\") " Oct 29 23:33:17.449716 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:17.449650 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bab23b35-8838-415c-90bf-aebf41448b58-kube-api-access-9hgh7\" (UniqueName: \"kubernetes.io/projected/bab23b35-8838-415c-90bf-aebf41448b58-kube-api-access-9hgh7\") pod \"bab23b35-8838-415c-90bf-aebf41448b58\" (UID: \"bab23b35-8838-415c-90bf-aebf41448b58\") " Oct 29 23:33:17.449716 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:17.449680 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bab23b35-8838-415c-90bf-aebf41448b58-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab23b35-8838-415c-90bf-aebf41448b58-openshift-service-ca\") pod \"bab23b35-8838-415c-90bf-aebf41448b58\" (UID: \"bab23b35-8838-415c-90bf-aebf41448b58\") " Oct 29 23:33:17.450074 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:17.450052 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab23b35-8838-415c-90bf-aebf41448b58-openshift-service-ca" pod "bab23b35-8838-415c-90bf-aebf41448b58" (UID: "bab23b35-8838-415c-90bf-aebf41448b58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:17.451891 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:17.451857 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab23b35-8838-415c-90bf-aebf41448b58-kube-api-access-9hgh7" pod "bab23b35-8838-415c-90bf-aebf41448b58" (UID: "bab23b35-8838-415c-90bf-aebf41448b58"). InnerVolumeSpecName "kube-api-access-9hgh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:17.452450 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:17.452428 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bab23b35-8838-415c-90bf-aebf41448b58-metrics-certs" pod "bab23b35-8838-415c-90bf-aebf41448b58" (UID: "bab23b35-8838-415c-90bf-aebf41448b58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:33:17.452529 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:17.452449 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bab23b35-8838-415c-90bf-aebf41448b58-maas-api-tls" pod "bab23b35-8838-415c-90bf-aebf41448b58" (UID: "bab23b35-8838-415c-90bf-aebf41448b58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:33:17.552910 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:17.552858 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hgh7\" (UniqueName: \"kubernetes.io/projected/bab23b35-8838-415c-90bf-aebf41448b58-kube-api-access-9hgh7\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:17.552910 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:17.552904 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab23b35-8838-415c-90bf-aebf41448b58-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:17.553194 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:17.552920 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bab23b35-8838-415c-90bf-aebf41448b58-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:17.553194 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:17.552932 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bab23b35-8838-415c-90bf-aebf41448b58-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:17.770855 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:17.770819 4821 generic.go:356] "Generic (PLEG): container finished" podID="bab23b35-8838-415c-90bf-aebf41448b58" containerID="b33053c0bc1933c80d4e767a974d2bd8dda3e82ca6e73778c81f61c5cd218838" exitCode=0 Oct 29 23:33:17.771074 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:17.770866 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp" Oct 29 23:33:17.771074 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:17.770896 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp" event={"ID":"bab23b35-8838-415c-90bf-aebf41448b58","Type":"ContainerDied","Data":"b33053c0bc1933c80d4e767a974d2bd8dda3e82ca6e73778c81f61c5cd218838"} Oct 29 23:33:17.771074 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:17.770956 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp" event={"ID":"bab23b35-8838-415c-90bf-aebf41448b58","Type":"ContainerDied","Data":"2651901ebcbb6cd2e60462e188a8666be1ef56ff3d8b05a7dd4b7acf1c14ba19"} Oct 29 23:33:17.771074 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:17.770970 4821 scope.go:122] "RemoveContainer" containerID="b33053c0bc1933c80d4e767a974d2bd8dda3e82ca6e73778c81f61c5cd218838" Oct 29 23:33:17.782109 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:17.782092 4821 scope.go:122] "RemoveContainer" containerID="b33053c0bc1933c80d4e767a974d2bd8dda3e82ca6e73778c81f61c5cd218838" Oct 29 23:33:17.782390 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:33:17.782365 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b33053c0bc1933c80d4e767a974d2bd8dda3e82ca6e73778c81f61c5cd218838\": container with ID starting with b33053c0bc1933c80d4e767a974d2bd8dda3e82ca6e73778c81f61c5cd218838 not found: ID does not exist" containerID="b33053c0bc1933c80d4e767a974d2bd8dda3e82ca6e73778c81f61c5cd218838" Oct 29 23:33:17.782446 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:17.782401 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33053c0bc1933c80d4e767a974d2bd8dda3e82ca6e73778c81f61c5cd218838"} err="failed to get container status \"b33053c0bc1933c80d4e767a974d2bd8dda3e82ca6e73778c81f61c5cd218838\": rpc error: code = NotFound desc = could not find container \"b33053c0bc1933c80d4e767a974d2bd8dda3e82ca6e73778c81f61c5cd218838\": container with ID starting with b33053c0bc1933c80d4e767a974d2bd8dda3e82ca6e73778c81f61c5cd218838 not found: ID does not exist" Oct 29 23:33:17.824181 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:17.824159 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp"] Oct 29 23:33:17.827110 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:17.827086 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-67363b-6bb6986cbb-626gp"] Oct 29 23:33:18.169449 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:18.169415 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bab23b35-8838-415c-90bf-aebf41448b58" path="/var/lib/kubelet/pods/bab23b35-8838-415c-90bf-aebf41448b58/volumes" Oct 29 23:33:20.008251 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.008163 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 23:33:20.079857 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.079824 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-podinfo\") pod \"9be68e75-df62-4a43-9ae9-20c04683f4c7\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " Oct 29 23:33:20.080057 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.079897 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-credential-socket\") pod \"9be68e75-df62-4a43-9ae9-20c04683f4c7\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " Oct 29 23:33:20.080057 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.079928 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9be68e75-df62-4a43-9ae9-20c04683f4c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9be68e75-df62-4a43-9ae9-20c04683f4c7-istiod-ca-cert\") pod \"9be68e75-df62-4a43-9ae9-20c04683f4c7\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " Oct 29 23:33:20.080057 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.079953 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-workload-socket\") pod \"9be68e75-df62-4a43-9ae9-20c04683f4c7\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " Oct 29 23:33:20.080057 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.080011 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-data\") pod \"9be68e75-df62-4a43-9ae9-20c04683f4c7\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " Oct 29 23:33:20.080350 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.080075 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-workload-certs\") pod \"9be68e75-df62-4a43-9ae9-20c04683f4c7\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " Oct 29 23:33:20.080350 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.080133 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-envoy\") pod \"9be68e75-df62-4a43-9ae9-20c04683f4c7\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " Oct 29 23:33:20.080350 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.080235 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-token\" (UniqueName: \"kubernetes.io/projected/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-token\") pod \"9be68e75-df62-4a43-9ae9-20c04683f4c7\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " Oct 29 23:33:20.080350 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.080278 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9be68e75-df62-4a43-9ae9-20c04683f4c7-kube-api-access-ngxpk\" (UniqueName: \"kubernetes.io/projected/9be68e75-df62-4a43-9ae9-20c04683f4c7-kube-api-access-ngxpk\") pod \"9be68e75-df62-4a43-9ae9-20c04683f4c7\" (UID: \"9be68e75-df62-4a43-9ae9-20c04683f4c7\") " Oct 29 23:33:20.080576 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.080387 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-workload-socket" pod "9be68e75-df62-4a43-9ae9-20c04683f4c7" (UID: "9be68e75-df62-4a43-9ae9-20c04683f4c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:20.080576 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.080491 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9be68e75-df62-4a43-9ae9-20c04683f4c7-istiod-ca-cert" pod "9be68e75-df62-4a43-9ae9-20c04683f4c7" (UID: "9be68e75-df62-4a43-9ae9-20c04683f4c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:20.080833 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.080811 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9be68e75-df62-4a43-9ae9-20c04683f4c7-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:20.081174 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.081060 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:20.081332 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.080830 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-credential-socket" pod "9be68e75-df62-4a43-9ae9-20c04683f4c7" (UID: "9be68e75-df62-4a43-9ae9-20c04683f4c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:20.081332 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.080855 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-data" pod "9be68e75-df62-4a43-9ae9-20c04683f4c7" (UID: "9be68e75-df62-4a43-9ae9-20c04683f4c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:20.081332 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.080911 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-workload-certs" pod "9be68e75-df62-4a43-9ae9-20c04683f4c7" (UID: "9be68e75-df62-4a43-9ae9-20c04683f4c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:20.082747 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.082721 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be68e75-df62-4a43-9ae9-20c04683f4c7-kube-api-access-ngxpk" pod "9be68e75-df62-4a43-9ae9-20c04683f4c7" (UID: "9be68e75-df62-4a43-9ae9-20c04683f4c7"). InnerVolumeSpecName "kube-api-access-ngxpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:20.082959 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.082932 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-envoy" pod "9be68e75-df62-4a43-9ae9-20c04683f4c7" (UID: "9be68e75-df62-4a43-9ae9-20c04683f4c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:20.083062 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.083044 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-podinfo" pod "9be68e75-df62-4a43-9ae9-20c04683f4c7" (UID: "9be68e75-df62-4a43-9ae9-20c04683f4c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:33:20.083163 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.083133 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-token" pod "9be68e75-df62-4a43-9ae9-20c04683f4c7" (UID: "9be68e75-df62-4a43-9ae9-20c04683f4c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:20.181795 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.181771 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:20.181795 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.181795 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:20.181957 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.181803 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:20.181957 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.181813 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngxpk\" (UniqueName: \"kubernetes.io/projected/9be68e75-df62-4a43-9ae9-20c04683f4c7-kube-api-access-ngxpk\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:20.181957 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.181821 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:20.181957 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.181829 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:20.181957 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.181836 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9be68e75-df62-4a43-9ae9-20c04683f4c7-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:20.793360 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.793325 4821 generic.go:356] "Generic (PLEG): container finished" podID="9be68e75-df62-4a43-9ae9-20c04683f4c7" containerID="7934b7a54f73d2c8220c99250e75365b6e5c194a793de9b3ca9125e554500004" exitCode=0 Oct 29 23:33:20.793550 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.793371 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" event={"ID":"9be68e75-df62-4a43-9ae9-20c04683f4c7","Type":"ContainerDied","Data":"7934b7a54f73d2c8220c99250e75365b6e5c194a793de9b3ca9125e554500004"} Oct 29 23:33:20.793550 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.793377 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" Oct 29 23:33:20.793550 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.793412 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4" event={"ID":"9be68e75-df62-4a43-9ae9-20c04683f4c7","Type":"ContainerDied","Data":"533b4ba493162a6275126c18e024be067a7b3986e9b2de7241d9f57100229c2e"} Oct 29 23:33:20.793550 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.793433 4821 scope.go:122] "RemoveContainer" containerID="7934b7a54f73d2c8220c99250e75365b6e5c194a793de9b3ca9125e554500004" Oct 29 23:33:20.803354 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.803337 4821 scope.go:122] "RemoveContainer" containerID="7934b7a54f73d2c8220c99250e75365b6e5c194a793de9b3ca9125e554500004" Oct 29 23:33:20.803671 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:33:20.803651 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7934b7a54f73d2c8220c99250e75365b6e5c194a793de9b3ca9125e554500004\": container with ID starting with 7934b7a54f73d2c8220c99250e75365b6e5c194a793de9b3ca9125e554500004 not found: ID does not exist" containerID="7934b7a54f73d2c8220c99250e75365b6e5c194a793de9b3ca9125e554500004" Oct 29 23:33:20.803718 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.803681 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7934b7a54f73d2c8220c99250e75365b6e5c194a793de9b3ca9125e554500004"} err="failed to get container status \"7934b7a54f73d2c8220c99250e75365b6e5c194a793de9b3ca9125e554500004\": rpc error: code = NotFound desc = could not find container \"7934b7a54f73d2c8220c99250e75365b6e5c194a793de9b3ca9125e554500004\": container with ID starting with 7934b7a54f73d2c8220c99250e75365b6e5c194a793de9b3ca9125e554500004 not found: ID does not exist" Oct 29 23:33:20.862847 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.862815 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4"] Oct 29 23:33:20.876235 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:20.876193 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e131af-openshift-default-c476c699c-dhws4"] Oct 29 23:33:21.177761 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:21.177719 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz" podUID="9809020b-a844-44fa-86af-6565434de12d" containerName="payload-processing" containerID="cri-o://dde4eba0ad1b9a2395f591735f4880250219ee3648b9ac2d98fcaf2c02c7adc5" gracePeriod=25 Oct 29 23:33:21.208431 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:21.208396 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt" podUID="9a821056-9bb0-4430-ab80-d6c44d3fa4f5" containerName="payload-pre-processing" containerID="cri-o://fe986ccc7ea53813b6f95d494d5f8b6eb50088ea9e4378031b7cb0ce83db9e06" gracePeriod=25 Oct 29 23:33:22.177190 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:22.177145 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9be68e75-df62-4a43-9ae9-20c04683f4c7" path="/var/lib/kubelet/pods/9be68e75-df62-4a43-9ae9-20c04683f4c7/volumes" Oct 29 23:33:23.883239 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:33:23.883187 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:33:24.158626 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:24.158546 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz" podUID="9809020b-a844-44fa-86af-6565434de12d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:33:24.158835 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:24.158791 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt" podUID="9a821056-9bb0-4430-ab80-d6c44d3fa4f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:33:25.359080 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.359036 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" containerID="cri-o://e230c235c18b884a72b4a6c9cc855193ba8086294b06f96112afedaa56a2ef34" gracePeriod=45 Oct 29 23:33:25.373390 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.373355 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" containerID="cri-o://91b87b818340ad5872492d15dd0124d96523708b9e2b15d866295e13b6f1df4e" gracePeriod=45 Oct 29 23:33:25.628595 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.628567 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 23:33:25.658210 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.658172 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 23:33:25.744311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.744277 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-model-cache\") pod \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\" (UID: \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\") " Oct 29 23:33:25.744518 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.744324 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-home\" (UniqueName: \"kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-home\") pod \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\" (UID: \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\") " Oct 29 23:33:25.744518 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.744392 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-dshm\") pod \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\" (UID: \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\") " Oct 29 23:33:25.744518 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.744438 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-kube-api-access-x7txf\" (UniqueName: \"kubernetes.io/projected/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-kube-api-access-x7txf\") pod \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\" (UID: \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\") " Oct 29 23:33:25.744518 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.744469 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e04ee6b-8e5b-4dc1-b001-184689f651e5-kube-api-access-bvv7c\" (UniqueName: \"kubernetes.io/projected/8e04ee6b-8e5b-4dc1-b001-184689f651e5-kube-api-access-bvv7c\") pod \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\" (UID: \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\") " Oct 29 23:33:25.744776 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.744542 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-tls-certs\") pod \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\" (UID: \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\") " Oct 29 23:33:25.744776 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.744583 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-tmp-dir\") pod \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\" (UID: \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\") " Oct 29 23:33:25.744776 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.744618 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-dshm\") pod \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\" (UID: \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\") " Oct 29 23:33:25.744776 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.744628 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-model-cache" pod "8e04ee6b-8e5b-4dc1-b001-184689f651e5" (UID: "8e04ee6b-8e5b-4dc1-b001-184689f651e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:25.744776 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.744659 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-home\" (UniqueName: \"kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-home\") pod \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\" (UID: \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\") " Oct 29 23:33:25.744776 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.744736 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-model-cache\") pod \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\" (UID: \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\") " Oct 29 23:33:25.744776 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.744764 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-tmp-dir\") pod \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\" (UID: \"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6\") " Oct 29 23:33:25.745147 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.744790 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e04ee6b-8e5b-4dc1-b001-184689f651e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e04ee6b-8e5b-4dc1-b001-184689f651e5-tls-certs\") pod \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\" (UID: \"8e04ee6b-8e5b-4dc1-b001-184689f651e5\") " Oct 29 23:33:25.745147 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.745112 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:25.745426 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.745400 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-home" pod "8e04ee6b-8e5b-4dc1-b001-184689f651e5" (UID: "8e04ee6b-8e5b-4dc1-b001-184689f651e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:25.745521 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.745407 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-home" pod "c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" (UID: "c436ecf5-3fdf-43ea-be7e-d58eaf7481b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:25.745623 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.745597 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-tmp-dir" pod "8e04ee6b-8e5b-4dc1-b001-184689f651e5" (UID: "8e04ee6b-8e5b-4dc1-b001-184689f651e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:25.745768 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.745747 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-tmp-dir" pod "c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" (UID: "c436ecf5-3fdf-43ea-be7e-d58eaf7481b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:25.745847 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.745827 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-model-cache" pod "c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" (UID: "c436ecf5-3fdf-43ea-be7e-d58eaf7481b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:25.747687 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.747658 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e04ee6b-8e5b-4dc1-b001-184689f651e5-kube-api-access-bvv7c" pod "8e04ee6b-8e5b-4dc1-b001-184689f651e5" (UID: "8e04ee6b-8e5b-4dc1-b001-184689f651e5"). InnerVolumeSpecName "kube-api-access-bvv7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:25.747854 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.747831 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e04ee6b-8e5b-4dc1-b001-184689f651e5-tls-certs" pod "8e04ee6b-8e5b-4dc1-b001-184689f651e5" (UID: "8e04ee6b-8e5b-4dc1-b001-184689f651e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:33:25.748377 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.748360 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-kube-api-access-x7txf" pod "c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" (UID: "c436ecf5-3fdf-43ea-be7e-d58eaf7481b6"). InnerVolumeSpecName "kube-api-access-x7txf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:25.748377 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.748360 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-tls-certs" pod "c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" (UID: "c436ecf5-3fdf-43ea-be7e-d58eaf7481b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:33:25.749587 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.749558 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-dshm" pod "c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" (UID: "c436ecf5-3fdf-43ea-be7e-d58eaf7481b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:25.750167 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.750148 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-dshm" pod "8e04ee6b-8e5b-4dc1-b001-184689f651e5" (UID: "8e04ee6b-8e5b-4dc1-b001-184689f651e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:25.823671 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.823632 4821 generic.go:356] "Generic (PLEG): container finished" podID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerID="91b87b818340ad5872492d15dd0124d96523708b9e2b15d866295e13b6f1df4e" exitCode=0 Oct 29 23:33:25.823867 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.823691 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" Oct 29 23:33:25.823867 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.823708 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" event={"ID":"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6","Type":"ContainerDied","Data":"91b87b818340ad5872492d15dd0124d96523708b9e2b15d866295e13b6f1df4e"} Oct 29 23:33:25.823867 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.823748 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx" event={"ID":"c436ecf5-3fdf-43ea-be7e-d58eaf7481b6","Type":"ContainerDied","Data":"349992c5575f141639db54f7d2a0aff701112953a6e464ae0efc8772fec65a60"} Oct 29 23:33:25.823867 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.823765 4821 scope.go:122] "RemoveContainer" containerID="91b87b818340ad5872492d15dd0124d96523708b9e2b15d866295e13b6f1df4e" Oct 29 23:33:25.827129 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.825497 4821 generic.go:356] "Generic (PLEG): container finished" podID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerID="e230c235c18b884a72b4a6c9cc855193ba8086294b06f96112afedaa56a2ef34" exitCode=0 Oct 29 23:33:25.827129 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.825946 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" Oct 29 23:33:25.827129 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.825576 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" event={"ID":"8e04ee6b-8e5b-4dc1-b001-184689f651e5","Type":"ContainerDied","Data":"e230c235c18b884a72b4a6c9cc855193ba8086294b06f96112afedaa56a2ef34"} Oct 29 23:33:25.827129 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.827084 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm" event={"ID":"8e04ee6b-8e5b-4dc1-b001-184689f651e5","Type":"ContainerDied","Data":"0859e3f69f22fc4c0e5ff74a4d36bedf45a6cf18e9789e89c70e82dbcf8c45f4"} Oct 29 23:33:25.837127 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.837015 4821 scope.go:122] "RemoveContainer" containerID="91b87b818340ad5872492d15dd0124d96523708b9e2b15d866295e13b6f1df4e" Oct 29 23:33:25.837491 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:33:25.837425 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91b87b818340ad5872492d15dd0124d96523708b9e2b15d866295e13b6f1df4e\": container with ID starting with 91b87b818340ad5872492d15dd0124d96523708b9e2b15d866295e13b6f1df4e not found: ID does not exist" containerID="91b87b818340ad5872492d15dd0124d96523708b9e2b15d866295e13b6f1df4e" Oct 29 23:33:25.837491 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.837454 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b87b818340ad5872492d15dd0124d96523708b9e2b15d866295e13b6f1df4e"} err="failed to get container status \"91b87b818340ad5872492d15dd0124d96523708b9e2b15d866295e13b6f1df4e\": rpc error: code = NotFound desc = could not find container \"91b87b818340ad5872492d15dd0124d96523708b9e2b15d866295e13b6f1df4e\": container with ID starting with 91b87b818340ad5872492d15dd0124d96523708b9e2b15d866295e13b6f1df4e not found: ID does not exist" Oct 29 23:33:25.837491 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.837471 4821 scope.go:122] "RemoveContainer" containerID="e230c235c18b884a72b4a6c9cc855193ba8086294b06f96112afedaa56a2ef34" Oct 29 23:33:25.845813 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.845790 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:25.845813 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.845811 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7txf\" (UniqueName: \"kubernetes.io/projected/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-kube-api-access-x7txf\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:25.845946 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.845821 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvv7c\" (UniqueName: \"kubernetes.io/projected/8e04ee6b-8e5b-4dc1-b001-184689f651e5-kube-api-access-bvv7c\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:25.845946 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.845830 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:25.845946 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.845838 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:25.845946 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.845845 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:25.845946 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.845852 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e04ee6b-8e5b-4dc1-b001-184689f651e5-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:25.845946 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.845860 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:25.845946 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.845867 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:25.845946 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.845874 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e04ee6b-8e5b-4dc1-b001-184689f651e5-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:25.845946 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.845881 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:25.846300 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.846285 4821 scope.go:122] "RemoveContainer" containerID="e230c235c18b884a72b4a6c9cc855193ba8086294b06f96112afedaa56a2ef34" Oct 29 23:33:25.846528 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:33:25.846511 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e230c235c18b884a72b4a6c9cc855193ba8086294b06f96112afedaa56a2ef34\": container with ID starting with e230c235c18b884a72b4a6c9cc855193ba8086294b06f96112afedaa56a2ef34 not found: ID does not exist" containerID="e230c235c18b884a72b4a6c9cc855193ba8086294b06f96112afedaa56a2ef34" Oct 29 23:33:25.846578 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.846535 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e230c235c18b884a72b4a6c9cc855193ba8086294b06f96112afedaa56a2ef34"} err="failed to get container status \"e230c235c18b884a72b4a6c9cc855193ba8086294b06f96112afedaa56a2ef34\": rpc error: code = NotFound desc = could not find container \"e230c235c18b884a72b4a6c9cc855193ba8086294b06f96112afedaa56a2ef34\": container with ID starting with e230c235c18b884a72b4a6c9cc855193ba8086294b06f96112afedaa56a2ef34 not found: ID does not exist" Oct 29 23:33:25.879536 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.879485 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm"] Oct 29 23:33:25.886526 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.886501 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-063b46/distinct-2-w4-063b46-kserve-7f485fc759-b65nm"] Oct 29 23:33:25.922061 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.922030 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx"] Oct 29 23:33:25.930654 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:25.930624 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-063b46/premium-w4-063b46-kserve-6c46d9cfcc-fp7fx"] Oct 29 23:33:26.169728 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:26.169637 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" path="/var/lib/kubelet/pods/8e04ee6b-8e5b-4dc1-b001-184689f651e5/volumes" Oct 29 23:33:26.170123 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:26.170102 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" path="/var/lib/kubelet/pods/c436ecf5-3fdf-43ea-be7e-d58eaf7481b6/volumes" Oct 29 23:33:30.795685 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:30.795650 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn"] Oct 29 23:33:30.808644 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:30.808611 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6"] Oct 29 23:33:34.157673 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:34.157631 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz" podUID="9809020b-a844-44fa-86af-6565434de12d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:33:34.158043 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:34.157695 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt" podUID="9a821056-9bb0-4430-ab80-d6c44d3fa4f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:33:41.487422 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.487385 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz" Oct 29 23:33:41.517756 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.517734 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt" Oct 29 23:33:41.607900 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.607867 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9809020b-a844-44fa-86af-6565434de12d-kube-api-access-lvbdd\" (UniqueName: \"kubernetes.io/projected/9809020b-a844-44fa-86af-6565434de12d-kube-api-access-lvbdd\") pod \"9809020b-a844-44fa-86af-6565434de12d\" (UID: \"9809020b-a844-44fa-86af-6565434de12d\") " Oct 29 23:33:41.608072 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.607979 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9809020b-a844-44fa-86af-6565434de12d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9809020b-a844-44fa-86af-6565434de12d-plugins-config-volume\") pod \"9809020b-a844-44fa-86af-6565434de12d\" (UID: \"9809020b-a844-44fa-86af-6565434de12d\") " Oct 29 23:33:41.608341 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.608316 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9809020b-a844-44fa-86af-6565434de12d-plugins-config-volume" pod "9809020b-a844-44fa-86af-6565434de12d" (UID: "9809020b-a844-44fa-86af-6565434de12d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:41.610068 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.610049 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9809020b-a844-44fa-86af-6565434de12d-kube-api-access-lvbdd" pod "9809020b-a844-44fa-86af-6565434de12d" (UID: "9809020b-a844-44fa-86af-6565434de12d"). InnerVolumeSpecName "kube-api-access-lvbdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:41.708940 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.708868 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a821056-9bb0-4430-ab80-d6c44d3fa4f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a821056-9bb0-4430-ab80-d6c44d3fa4f5-plugins-config-volume\") pod \"9a821056-9bb0-4430-ab80-d6c44d3fa4f5\" (UID: \"9a821056-9bb0-4430-ab80-d6c44d3fa4f5\") " Oct 29 23:33:41.709091 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.709008 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a821056-9bb0-4430-ab80-d6c44d3fa4f5-kube-api-access-lsqcf\" (UniqueName: \"kubernetes.io/projected/9a821056-9bb0-4430-ab80-d6c44d3fa4f5-kube-api-access-lsqcf\") pod \"9a821056-9bb0-4430-ab80-d6c44d3fa4f5\" (UID: \"9a821056-9bb0-4430-ab80-d6c44d3fa4f5\") " Oct 29 23:33:41.709311 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.709286 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a821056-9bb0-4430-ab80-d6c44d3fa4f5-plugins-config-volume" pod "9a821056-9bb0-4430-ab80-d6c44d3fa4f5" (UID: "9a821056-9bb0-4430-ab80-d6c44d3fa4f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:41.709367 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.709325 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvbdd\" (UniqueName: \"kubernetes.io/projected/9809020b-a844-44fa-86af-6565434de12d-kube-api-access-lvbdd\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:41.709367 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.709340 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9809020b-a844-44fa-86af-6565434de12d-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:41.711173 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.711146 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a821056-9bb0-4430-ab80-d6c44d3fa4f5-kube-api-access-lsqcf" pod "9a821056-9bb0-4430-ab80-d6c44d3fa4f5" (UID: "9a821056-9bb0-4430-ab80-d6c44d3fa4f5"). InnerVolumeSpecName "kube-api-access-lsqcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:41.810652 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.810619 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsqcf\" (UniqueName: \"kubernetes.io/projected/9a821056-9bb0-4430-ab80-d6c44d3fa4f5-kube-api-access-lsqcf\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:41.810652 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.810649 4821 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a821056-9bb0-4430-ab80-d6c44d3fa4f5-plugins-config-volume\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:41.918594 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.918559 4821 generic.go:356] "Generic (PLEG): container finished" podID="9a821056-9bb0-4430-ab80-d6c44d3fa4f5" containerID="fe986ccc7ea53813b6f95d494d5f8b6eb50088ea9e4378031b7cb0ce83db9e06" exitCode=0 Oct 29 23:33:41.918789 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.918602 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt" Oct 29 23:33:41.918789 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.918650 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt" event={"ID":"9a821056-9bb0-4430-ab80-d6c44d3fa4f5","Type":"ContainerDied","Data":"fe986ccc7ea53813b6f95d494d5f8b6eb50088ea9e4378031b7cb0ce83db9e06"} Oct 29 23:33:41.918789 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.918703 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt" event={"ID":"9a821056-9bb0-4430-ab80-d6c44d3fa4f5","Type":"ContainerDied","Data":"56c007ae28bff4a54199eaf083449dc9bd7af1f22cc23e3aac38604750584136"} Oct 29 23:33:41.918789 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.918733 4821 scope.go:122] "RemoveContainer" containerID="fe986ccc7ea53813b6f95d494d5f8b6eb50088ea9e4378031b7cb0ce83db9e06" Oct 29 23:33:41.920079 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.920057 4821 generic.go:356] "Generic (PLEG): container finished" podID="9809020b-a844-44fa-86af-6565434de12d" containerID="dde4eba0ad1b9a2395f591735f4880250219ee3648b9ac2d98fcaf2c02c7adc5" exitCode=0 Oct 29 23:33:41.920193 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.920090 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz" event={"ID":"9809020b-a844-44fa-86af-6565434de12d","Type":"ContainerDied","Data":"dde4eba0ad1b9a2395f591735f4880250219ee3648b9ac2d98fcaf2c02c7adc5"} Oct 29 23:33:41.920193 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.920114 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz" Oct 29 23:33:41.920193 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.920144 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz" event={"ID":"9809020b-a844-44fa-86af-6565434de12d","Type":"ContainerDied","Data":"06013545c61dcef6ebaf9ff520ccaba551305e13622c10172235908b1d0dd011"} Oct 29 23:33:41.929972 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.929950 4821 scope.go:122] "RemoveContainer" containerID="fe986ccc7ea53813b6f95d494d5f8b6eb50088ea9e4378031b7cb0ce83db9e06" Oct 29 23:33:41.930310 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:33:41.930251 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe986ccc7ea53813b6f95d494d5f8b6eb50088ea9e4378031b7cb0ce83db9e06\": container with ID starting with fe986ccc7ea53813b6f95d494d5f8b6eb50088ea9e4378031b7cb0ce83db9e06 not found: ID does not exist" containerID="fe986ccc7ea53813b6f95d494d5f8b6eb50088ea9e4378031b7cb0ce83db9e06" Oct 29 23:33:41.930310 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.930289 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe986ccc7ea53813b6f95d494d5f8b6eb50088ea9e4378031b7cb0ce83db9e06"} err="failed to get container status \"fe986ccc7ea53813b6f95d494d5f8b6eb50088ea9e4378031b7cb0ce83db9e06\": rpc error: code = NotFound desc = could not find container \"fe986ccc7ea53813b6f95d494d5f8b6eb50088ea9e4378031b7cb0ce83db9e06\": container with ID starting with fe986ccc7ea53813b6f95d494d5f8b6eb50088ea9e4378031b7cb0ce83db9e06 not found: ID does not exist" Oct 29 23:33:41.930310 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.930307 4821 scope.go:122] "RemoveContainer" containerID="dde4eba0ad1b9a2395f591735f4880250219ee3648b9ac2d98fcaf2c02c7adc5" Oct 29 23:33:41.939125 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.939107 4821 scope.go:122] "RemoveContainer" containerID="dde4eba0ad1b9a2395f591735f4880250219ee3648b9ac2d98fcaf2c02c7adc5" Oct 29 23:33:41.939396 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:33:41.939376 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dde4eba0ad1b9a2395f591735f4880250219ee3648b9ac2d98fcaf2c02c7adc5\": container with ID starting with dde4eba0ad1b9a2395f591735f4880250219ee3648b9ac2d98fcaf2c02c7adc5 not found: ID does not exist" containerID="dde4eba0ad1b9a2395f591735f4880250219ee3648b9ac2d98fcaf2c02c7adc5" Oct 29 23:33:41.939497 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.939414 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dde4eba0ad1b9a2395f591735f4880250219ee3648b9ac2d98fcaf2c02c7adc5"} err="failed to get container status \"dde4eba0ad1b9a2395f591735f4880250219ee3648b9ac2d98fcaf2c02c7adc5\": rpc error: code = NotFound desc = could not find container \"dde4eba0ad1b9a2395f591735f4880250219ee3648b9ac2d98fcaf2c02c7adc5\": container with ID starting with dde4eba0ad1b9a2395f591735f4880250219ee3648b9ac2d98fcaf2c02c7adc5 not found: ID does not exist" Oct 29 23:33:41.971668 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.971610 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt"] Oct 29 23:33:41.984594 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:41.984573 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-063b46-7f4c5465b7-txdkt"] Oct 29 23:33:42.010102 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:42.010072 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz"] Oct 29 23:33:42.013209 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:42.013175 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-063b46-754b7797fb-qg6wz"] Oct 29 23:33:42.167813 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:42.167781 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9809020b-a844-44fa-86af-6565434de12d" path="/var/lib/kubelet/pods/9809020b-a844-44fa-86af-6565434de12d/volumes" Oct 29 23:33:42.168160 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:42.168148 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a821056-9bb0-4430-ab80-d6c44d3fa4f5" path="/var/lib/kubelet/pods/9a821056-9bb0-4430-ab80-d6c44d3fa4f5/volumes" Oct 29 23:33:45.541367 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:45.541329 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb"] Oct 29 23:33:45.541877 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:45.541584 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" podUID="ea79c88e-57ac-4f8f-a392-bf31d397cae8" containerName="istio-proxy" containerID="cri-o://bd48930866127a1b2060e2d12207b6ce7e34c63f1e1f1ff9f9cbce3544fb1239" gracePeriod=30 Oct 29 23:33:45.817538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:45.817443 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" containerID="cri-o://1f87c1f07c414a80463890155eb3a5dc315ad4848538f803f3e7965a597766f7" gracePeriod=45 Oct 29 23:33:45.828665 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:45.828624 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" containerID="cri-o://321f062041e5c67688472c8b1eaa42514a1862591aa1b37d389ea82c74470f21" gracePeriod=45 Oct 29 23:33:45.945772 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:45.945739 4821 generic.go:356] "Generic (PLEG): container finished" podID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerID="1f87c1f07c414a80463890155eb3a5dc315ad4848538f803f3e7965a597766f7" exitCode=0 Oct 29 23:33:45.945897 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:45.945804 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" event={"ID":"28f5dfd6-1e9c-447d-ac89-82393448335d","Type":"ContainerDied","Data":"1f87c1f07c414a80463890155eb3a5dc315ad4848538f803f3e7965a597766f7"} Oct 29 23:33:45.991565 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:45.991506 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": dial tcp 10.133.0.47:8000: connect: connection refused" Oct 29 23:33:46.075073 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.075015 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 23:33:46.103174 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.103149 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 23:33:46.151801 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.151763 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-model-cache\") pod \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\" (UID: \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\") " Oct 29 23:33:46.151801 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.151799 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-model-cache\") pod \"28f5dfd6-1e9c-447d-ac89-82393448335d\" (UID: \"28f5dfd6-1e9c-447d-ac89-82393448335d\") " Oct 29 23:33:46.152063 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.151846 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeacdd0d-a01d-498c-bf12-f384dc79d57f-kube-api-access-v89r2\" (UniqueName: \"kubernetes.io/projected/eeacdd0d-a01d-498c-bf12-f384dc79d57f-kube-api-access-v89r2\") pod \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\" (UID: \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\") " Oct 29 23:33:46.152063 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.151874 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-tmp-dir\") pod \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\" (UID: \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\") " Oct 29 23:33:46.152063 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.151908 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-dshm\") pod \"28f5dfd6-1e9c-447d-ac89-82393448335d\" (UID: \"28f5dfd6-1e9c-447d-ac89-82393448335d\") " Oct 29 23:33:46.152063 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.151941 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-home\" (UniqueName: \"kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-home\") pod \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\" (UID: \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\") " Oct 29 23:33:46.152063 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.151966 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-home\" (UniqueName: \"kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-home\") pod \"28f5dfd6-1e9c-447d-ac89-82393448335d\" (UID: \"28f5dfd6-1e9c-447d-ac89-82393448335d\") " Oct 29 23:33:46.152063 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.152050 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-tmp-dir\") pod \"28f5dfd6-1e9c-447d-ac89-82393448335d\" (UID: \"28f5dfd6-1e9c-447d-ac89-82393448335d\") " Oct 29 23:33:46.152445 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.152074 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-model-cache" pod "eeacdd0d-a01d-498c-bf12-f384dc79d57f" (UID: "eeacdd0d-a01d-498c-bf12-f384dc79d57f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:46.152445 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.152092 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28f5dfd6-1e9c-447d-ac89-82393448335d-tls-certs\" (UniqueName: \"kubernetes.io/secret/28f5dfd6-1e9c-447d-ac89-82393448335d-tls-certs\") pod \"28f5dfd6-1e9c-447d-ac89-82393448335d\" (UID: \"28f5dfd6-1e9c-447d-ac89-82393448335d\") " Oct 29 23:33:46.152445 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.152125 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28f5dfd6-1e9c-447d-ac89-82393448335d-kube-api-access-kprln\" (UniqueName: \"kubernetes.io/projected/28f5dfd6-1e9c-447d-ac89-82393448335d-kube-api-access-kprln\") pod \"28f5dfd6-1e9c-447d-ac89-82393448335d\" (UID: \"28f5dfd6-1e9c-447d-ac89-82393448335d\") " Oct 29 23:33:46.152445 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.152143 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-model-cache" pod "28f5dfd6-1e9c-447d-ac89-82393448335d" (UID: "28f5dfd6-1e9c-447d-ac89-82393448335d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:46.152445 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.152176 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-dshm\") pod \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\" (UID: \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\") " Oct 29 23:33:46.152445 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.152296 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-home" pod "eeacdd0d-a01d-498c-bf12-f384dc79d57f" (UID: "eeacdd0d-a01d-498c-bf12-f384dc79d57f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:46.152445 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.152415 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-tmp-dir" pod "eeacdd0d-a01d-498c-bf12-f384dc79d57f" (UID: "eeacdd0d-a01d-498c-bf12-f384dc79d57f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:46.152445 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.152427 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-home" pod "28f5dfd6-1e9c-447d-ac89-82393448335d" (UID: "28f5dfd6-1e9c-447d-ac89-82393448335d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:46.153053 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.152468 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeacdd0d-a01d-498c-bf12-f384dc79d57f-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeacdd0d-a01d-498c-bf12-f384dc79d57f-tls-certs\") pod \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\" (UID: \"eeacdd0d-a01d-498c-bf12-f384dc79d57f\") " Oct 29 23:33:46.153053 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.152627 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-tmp-dir" pod "28f5dfd6-1e9c-447d-ac89-82393448335d" (UID: "28f5dfd6-1e9c-447d-ac89-82393448335d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:46.153053 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.152829 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:46.153053 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.152848 4821 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-home\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:46.153053 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.152861 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:46.153053 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.152881 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:46.153053 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.152897 4821 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-model-cache\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:46.153053 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.152907 4821 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-tmp-dir\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:46.155461 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.154776 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeacdd0d-a01d-498c-bf12-f384dc79d57f-kube-api-access-v89r2" pod "eeacdd0d-a01d-498c-bf12-f384dc79d57f" (UID: "eeacdd0d-a01d-498c-bf12-f384dc79d57f"). InnerVolumeSpecName "kube-api-access-v89r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:46.155461 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.154978 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-dshm" pod "eeacdd0d-a01d-498c-bf12-f384dc79d57f" (UID: "eeacdd0d-a01d-498c-bf12-f384dc79d57f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:46.155461 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.155103 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f5dfd6-1e9c-447d-ac89-82393448335d-kube-api-access-kprln" pod "28f5dfd6-1e9c-447d-ac89-82393448335d" (UID: "28f5dfd6-1e9c-447d-ac89-82393448335d"). InnerVolumeSpecName "kube-api-access-kprln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:46.155461 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.155393 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeacdd0d-a01d-498c-bf12-f384dc79d57f-tls-certs" pod "eeacdd0d-a01d-498c-bf12-f384dc79d57f" (UID: "eeacdd0d-a01d-498c-bf12-f384dc79d57f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:33:46.155687 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.155481 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-dshm" pod "28f5dfd6-1e9c-447d-ac89-82393448335d" (UID: "28f5dfd6-1e9c-447d-ac89-82393448335d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:46.156235 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.156099 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28f5dfd6-1e9c-447d-ac89-82393448335d-tls-certs" pod "28f5dfd6-1e9c-447d-ac89-82393448335d" (UID: "28f5dfd6-1e9c-447d-ac89-82393448335d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:33:46.253330 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.253298 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v89r2\" (UniqueName: \"kubernetes.io/projected/eeacdd0d-a01d-498c-bf12-f384dc79d57f-kube-api-access-v89r2\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:46.253330 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.253326 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28f5dfd6-1e9c-447d-ac89-82393448335d-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:46.253330 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.253336 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28f5dfd6-1e9c-447d-ac89-82393448335d-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:46.253574 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.253345 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kprln\" (UniqueName: \"kubernetes.io/projected/28f5dfd6-1e9c-447d-ac89-82393448335d-kube-api-access-kprln\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:46.253574 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.253353 4821 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeacdd0d-a01d-498c-bf12-f384dc79d57f-dshm\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:46.253574 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.253361 4821 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeacdd0d-a01d-498c-bf12-f384dc79d57f-tls-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:46.953210 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.953171 4821 generic.go:356] "Generic (PLEG): container finished" podID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerID="321f062041e5c67688472c8b1eaa42514a1862591aa1b37d389ea82c74470f21" exitCode=0 Oct 29 23:33:46.953632 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.953230 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" Oct 29 23:33:46.953632 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.953266 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" event={"ID":"eeacdd0d-a01d-498c-bf12-f384dc79d57f","Type":"ContainerDied","Data":"321f062041e5c67688472c8b1eaa42514a1862591aa1b37d389ea82c74470f21"} Oct 29 23:33:46.953632 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.953307 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6" event={"ID":"eeacdd0d-a01d-498c-bf12-f384dc79d57f","Type":"ContainerDied","Data":"be38575c459b81d015ad3849406a825318e3f1b73787c648071aa77c090f7734"} Oct 29 23:33:46.953632 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.953327 4821 scope.go:122] "RemoveContainer" containerID="321f062041e5c67688472c8b1eaa42514a1862591aa1b37d389ea82c74470f21" Oct 29 23:33:46.955069 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.955043 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" event={"ID":"28f5dfd6-1e9c-447d-ac89-82393448335d","Type":"ContainerDied","Data":"e329d06a76eaab17cae62d2b1af2e0c76ceb093dd3fc3b99c41e0ce4ea4b130b"} Oct 29 23:33:46.955069 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.955065 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" Oct 29 23:33:46.963133 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.963115 4821 scope.go:122] "RemoveContainer" containerID="321f062041e5c67688472c8b1eaa42514a1862591aa1b37d389ea82c74470f21" Oct 29 23:33:46.963400 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:33:46.963381 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"321f062041e5c67688472c8b1eaa42514a1862591aa1b37d389ea82c74470f21\": container with ID starting with 321f062041e5c67688472c8b1eaa42514a1862591aa1b37d389ea82c74470f21 not found: ID does not exist" containerID="321f062041e5c67688472c8b1eaa42514a1862591aa1b37d389ea82c74470f21" Oct 29 23:33:46.963495 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.963418 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321f062041e5c67688472c8b1eaa42514a1862591aa1b37d389ea82c74470f21"} err="failed to get container status \"321f062041e5c67688472c8b1eaa42514a1862591aa1b37d389ea82c74470f21\": rpc error: code = NotFound desc = could not find container \"321f062041e5c67688472c8b1eaa42514a1862591aa1b37d389ea82c74470f21\": container with ID starting with 321f062041e5c67688472c8b1eaa42514a1862591aa1b37d389ea82c74470f21 not found: ID does not exist" Oct 29 23:33:46.963495 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:46.963436 4821 scope.go:122] "RemoveContainer" containerID="1f87c1f07c414a80463890155eb3a5dc315ad4848538f803f3e7965a597766f7" Oct 29 23:33:47.009775 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:47.009742 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6"] Oct 29 23:33:47.021780 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:47.021751 4821 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:33:47.022808 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:47.022788 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-08e15d/simulator-w0-08e15d-kserve-7d75cc98bb-brwp6"] Oct 29 23:33:47.049621 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:47.049596 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn"] Oct 29 23:33:47.052429 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:47.052405 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-08e15d/premium-w0-08e15d-kserve-7cbc4d9bc-frwmn"] Oct 29 23:33:48.168733 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:48.168698 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" path="/var/lib/kubelet/pods/28f5dfd6-1e9c-447d-ac89-82393448335d/volumes" Oct 29 23:33:48.169241 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:48.169090 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" path="/var/lib/kubelet/pods/eeacdd0d-a01d-498c-bf12-f384dc79d57f/volumes" Oct 29 23:33:50.787174 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.787146 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 23:33:50.894499 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.894470 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-podinfo\") pod \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " Oct 29 23:33:50.894690 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.894515 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-data\") pod \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " Oct 29 23:33:50.894690 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.894577 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-credential-socket\") pod \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " Oct 29 23:33:50.894819 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.894703 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea79c88e-57ac-4f8f-a392-bf31d397cae8-kube-api-access-tb9wk\" (UniqueName: \"kubernetes.io/projected/ea79c88e-57ac-4f8f-a392-bf31d397cae8-kube-api-access-tb9wk\") pod \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " Oct 29 23:33:50.894819 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.894751 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-token\" (UniqueName: \"kubernetes.io/projected/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-token\") pod \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " Oct 29 23:33:50.894819 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.894794 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-workload-socket\") pod \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " Oct 29 23:33:50.894993 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.894824 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istiod-ca-cert\") pod \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " Oct 29 23:33:50.894993 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.894866 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-workload-certs\") pod \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " Oct 29 23:33:50.894993 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.894877 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-data" pod "ea79c88e-57ac-4f8f-a392-bf31d397cae8" (UID: "ea79c88e-57ac-4f8f-a392-bf31d397cae8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:50.894993 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.894895 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-credential-socket" pod "ea79c88e-57ac-4f8f-a392-bf31d397cae8" (UID: "ea79c88e-57ac-4f8f-a392-bf31d397cae8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:50.894993 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.894901 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-envoy\") pod \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\" (UID: \"ea79c88e-57ac-4f8f-a392-bf31d397cae8\") " Oct 29 23:33:50.895300 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.895036 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-workload-socket" pod "ea79c88e-57ac-4f8f-a392-bf31d397cae8" (UID: "ea79c88e-57ac-4f8f-a392-bf31d397cae8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:50.895300 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.895258 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-workload-certs" pod "ea79c88e-57ac-4f8f-a392-bf31d397cae8" (UID: "ea79c88e-57ac-4f8f-a392-bf31d397cae8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:50.895300 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.895261 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:50.895300 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.895287 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istiod-ca-cert" pod "ea79c88e-57ac-4f8f-a392-bf31d397cae8" (UID: "ea79c88e-57ac-4f8f-a392-bf31d397cae8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:50.895482 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.895301 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:50.895482 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.895317 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:50.897261 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.897235 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-envoy" pod "ea79c88e-57ac-4f8f-a392-bf31d397cae8" (UID: "ea79c88e-57ac-4f8f-a392-bf31d397cae8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:33:50.897396 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.897317 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-token" pod "ea79c88e-57ac-4f8f-a392-bf31d397cae8" (UID: "ea79c88e-57ac-4f8f-a392-bf31d397cae8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:50.897466 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.897434 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-podinfo" pod "ea79c88e-57ac-4f8f-a392-bf31d397cae8" (UID: "ea79c88e-57ac-4f8f-a392-bf31d397cae8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:33:50.897565 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.897479 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea79c88e-57ac-4f8f-a392-bf31d397cae8-kube-api-access-tb9wk" pod "ea79c88e-57ac-4f8f-a392-bf31d397cae8" (UID: "ea79c88e-57ac-4f8f-a392-bf31d397cae8"). InnerVolumeSpecName "kube-api-access-tb9wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:50.978657 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.978623 4821 generic.go:356] "Generic (PLEG): container finished" podID="ea79c88e-57ac-4f8f-a392-bf31d397cae8" containerID="bd48930866127a1b2060e2d12207b6ce7e34c63f1e1f1ff9f9cbce3544fb1239" exitCode=0 Oct 29 23:33:50.978797 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.978675 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" Oct 29 23:33:50.978797 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.978701 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" event={"ID":"ea79c88e-57ac-4f8f-a392-bf31d397cae8","Type":"ContainerDied","Data":"bd48930866127a1b2060e2d12207b6ce7e34c63f1e1f1ff9f9cbce3544fb1239"} Oct 29 23:33:50.978797 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.978754 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb" event={"ID":"ea79c88e-57ac-4f8f-a392-bf31d397cae8","Type":"ContainerDied","Data":"20faf028eecca7cfdfc0c41bbda5d9b578b7e72e098ab78c897f0036a910ffbf"} Oct 29 23:33:50.978797 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.978779 4821 scope.go:122] "RemoveContainer" containerID="bd48930866127a1b2060e2d12207b6ce7e34c63f1e1f1ff9f9cbce3544fb1239" Oct 29 23:33:50.992719 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.992702 4821 scope.go:122] "RemoveContainer" containerID="bd48930866127a1b2060e2d12207b6ce7e34c63f1e1f1ff9f9cbce3544fb1239" Oct 29 23:33:50.992990 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:33:50.992967 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd48930866127a1b2060e2d12207b6ce7e34c63f1e1f1ff9f9cbce3544fb1239\": container with ID starting with bd48930866127a1b2060e2d12207b6ce7e34c63f1e1f1ff9f9cbce3544fb1239 not found: ID does not exist" containerID="bd48930866127a1b2060e2d12207b6ce7e34c63f1e1f1ff9f9cbce3544fb1239" Oct 29 23:33:50.993059 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.992997 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd48930866127a1b2060e2d12207b6ce7e34c63f1e1f1ff9f9cbce3544fb1239"} err="failed to get container status \"bd48930866127a1b2060e2d12207b6ce7e34c63f1e1f1ff9f9cbce3544fb1239\": rpc error: code = NotFound desc = could not find container \"bd48930866127a1b2060e2d12207b6ce7e34c63f1e1f1ff9f9cbce3544fb1239\": container with ID starting with bd48930866127a1b2060e2d12207b6ce7e34c63f1e1f1ff9f9cbce3544fb1239 not found: ID does not exist" Oct 29 23:33:50.996736 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.996709 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:50.996826 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.996738 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb9wk\" (UniqueName: \"kubernetes.io/projected/ea79c88e-57ac-4f8f-a392-bf31d397cae8-kube-api-access-tb9wk\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:50.996826 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.996748 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:50.996826 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.996757 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:50.996826 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.996770 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:50.996826 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:50.996778 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea79c88e-57ac-4f8f-a392-bf31d397cae8-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:51.031855 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:51.031827 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb"] Oct 29 23:33:51.035879 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:51.035856 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-063b46-openshift-default-799d8d64cf-87lwb"] Oct 29 23:33:52.168947 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:52.168911 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea79c88e-57ac-4f8f-a392-bf31d397cae8" path="/var/lib/kubelet/pods/ea79c88e-57ac-4f8f-a392-bf31d397cae8/volumes" Oct 29 23:33:53.887473 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:33:53.887420 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:33:54.964498 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:54.964461 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9"] Oct 29 23:33:54.964969 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:54.964710 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9" podUID="9a799d83-b2e3-48c2-bb3a-acbffdaee0d2" containerName="maas-api" containerID="cri-o://711055e310e0f6037c0dea470743fbe9094df1dfc0685991e80255ced76a2bbc" gracePeriod=30 Oct 29 23:33:55.238651 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:55.238624 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9" Oct 29 23:33:55.334075 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:55.334046 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-maas-api-tls\") pod \"9a799d83-b2e3-48c2-bb3a-acbffdaee0d2\" (UID: \"9a799d83-b2e3-48c2-bb3a-acbffdaee0d2\") " Oct 29 23:33:55.334284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:55.334117 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-openshift-service-ca\") pod \"9a799d83-b2e3-48c2-bb3a-acbffdaee0d2\" (UID: \"9a799d83-b2e3-48c2-bb3a-acbffdaee0d2\") " Oct 29 23:33:55.334284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:55.334169 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-kube-api-access-sffpx\" (UniqueName: \"kubernetes.io/projected/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-kube-api-access-sffpx\") pod \"9a799d83-b2e3-48c2-bb3a-acbffdaee0d2\" (UID: \"9a799d83-b2e3-48c2-bb3a-acbffdaee0d2\") " Oct 29 23:33:55.334284 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:55.334220 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-metrics-certs\") pod \"9a799d83-b2e3-48c2-bb3a-acbffdaee0d2\" (UID: \"9a799d83-b2e3-48c2-bb3a-acbffdaee0d2\") " Oct 29 23:33:55.334514 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:55.334480 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-openshift-service-ca" pod "9a799d83-b2e3-48c2-bb3a-acbffdaee0d2" (UID: "9a799d83-b2e3-48c2-bb3a-acbffdaee0d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:33:55.336520 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:55.336490 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-kube-api-access-sffpx" pod "9a799d83-b2e3-48c2-bb3a-acbffdaee0d2" (UID: "9a799d83-b2e3-48c2-bb3a-acbffdaee0d2"). InnerVolumeSpecName "kube-api-access-sffpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:33:55.336625 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:55.336520 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-maas-api-tls" pod "9a799d83-b2e3-48c2-bb3a-acbffdaee0d2" (UID: "9a799d83-b2e3-48c2-bb3a-acbffdaee0d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:33:55.336625 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:55.336541 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-metrics-certs" pod "9a799d83-b2e3-48c2-bb3a-acbffdaee0d2" (UID: "9a799d83-b2e3-48c2-bb3a-acbffdaee0d2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:33:55.435697 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:55.435669 4821 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-openshift-service-ca\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:55.435697 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:55.435694 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sffpx\" (UniqueName: \"kubernetes.io/projected/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-kube-api-access-sffpx\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:55.435697 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:55.435705 4821 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-metrics-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:55.435950 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:55.435716 4821 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2-maas-api-tls\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:33:56.009434 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:56.009399 4821 generic.go:356] "Generic (PLEG): container finished" podID="9a799d83-b2e3-48c2-bb3a-acbffdaee0d2" containerID="711055e310e0f6037c0dea470743fbe9094df1dfc0685991e80255ced76a2bbc" exitCode=0 Oct 29 23:33:56.009928 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:56.009451 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9" Oct 29 23:33:56.009928 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:56.009484 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9" event={"ID":"9a799d83-b2e3-48c2-bb3a-acbffdaee0d2","Type":"ContainerDied","Data":"711055e310e0f6037c0dea470743fbe9094df1dfc0685991e80255ced76a2bbc"} Oct 29 23:33:56.009928 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:56.009534 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9" event={"ID":"9a799d83-b2e3-48c2-bb3a-acbffdaee0d2","Type":"ContainerDied","Data":"ff49aa8abb6f27ec7025db45db2fa49961b4e7ed70b2ba52c0a44d817b259b62"} Oct 29 23:33:56.009928 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:56.009562 4821 scope.go:122] "RemoveContainer" containerID="711055e310e0f6037c0dea470743fbe9094df1dfc0685991e80255ced76a2bbc" Oct 29 23:33:56.020183 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:56.020164 4821 scope.go:122] "RemoveContainer" containerID="711055e310e0f6037c0dea470743fbe9094df1dfc0685991e80255ced76a2bbc" Oct 29 23:33:56.020504 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:33:56.020485 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"711055e310e0f6037c0dea470743fbe9094df1dfc0685991e80255ced76a2bbc\": container with ID starting with 711055e310e0f6037c0dea470743fbe9094df1dfc0685991e80255ced76a2bbc not found: ID does not exist" containerID="711055e310e0f6037c0dea470743fbe9094df1dfc0685991e80255ced76a2bbc" Oct 29 23:33:56.020574 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:56.020517 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"711055e310e0f6037c0dea470743fbe9094df1dfc0685991e80255ced76a2bbc"} err="failed to get container status \"711055e310e0f6037c0dea470743fbe9094df1dfc0685991e80255ced76a2bbc\": rpc error: code = NotFound desc = could not find container \"711055e310e0f6037c0dea470743fbe9094df1dfc0685991e80255ced76a2bbc\": container with ID starting with 711055e310e0f6037c0dea470743fbe9094df1dfc0685991e80255ced76a2bbc not found: ID does not exist" Oct 29 23:33:56.064076 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:56.064044 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9"] Oct 29 23:33:56.077515 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:56.077482 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-08e15d-698ffcd666-2w9h9"] Oct 29 23:33:56.174421 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:33:56.174381 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a799d83-b2e3-48c2-bb3a-acbffdaee0d2" path="/var/lib/kubelet/pods/9a799d83-b2e3-48c2-bb3a-acbffdaee0d2/volumes" Oct 29 23:34:03.863088 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:03.863046 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dd5b9556c-7gmf5_37c1f92f-3914-412b-a7aa-0cc023dce2c0/authorino/0.log" Oct 29 23:34:04.386070 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:04.386041 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-n8p4p_47662ab4-2668-46cd-9e6b-bad98897c972/manager/0.log" Oct 29 23:34:04.585798 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:04.585770 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-rvnlb_4d5895b2-39aa-489e-8c3b-f9bf15b1a58a/kuadrant-console-plugin/0.log" Oct 29 23:34:05.691673 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:05.691640 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-khfmc_28d10fb8-128f-44db-b810-48a734f838c5/manager/0.log" Oct 29 23:34:06.706870 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:06.706843 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-j26l4_8d65af4a-3d99-4d72-94ce-cd7d5aee4182/manager/0.log" Oct 29 23:34:07.119591 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:07.119555 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rx2tt_e781f3f8-7c1e-4f60-b8a7-236e97928c29/manager/0.log" Oct 29 23:34:07.701273 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:07.701233 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-2gc8z_9cacdb56-27c0-4b69-ae05-8a9d2344972d/server/0.log" Oct 29 23:34:08.853596 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:08.853567 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dd5b9556c-7gmf5_37c1f92f-3914-412b-a7aa-0cc023dce2c0/authorino/0.log" Oct 29 23:34:14.030005 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:14.029967 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4471f343-bdxww"] Oct 29 23:34:14.032876 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:14.032848 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4471f343-bdxww"] Oct 29 23:34:14.167290 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:14.167253 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="609b76dc-c376-41a4-8eb8-43267cfa3f6e" path="/var/lib/kubelet/pods/609b76dc-c376-41a4-8eb8-43267cfa3f6e/volumes" Oct 29 23:34:19.500473 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:19.500439 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-j26l4_8d65af4a-3d99-4d72-94ce-cd7d5aee4182/manager/0.log" Oct 29 23:34:19.924715 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:19.924684 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rx2tt_e781f3f8-7c1e-4f60-b8a7-236e97928c29/manager/0.log" Oct 29 23:34:20.507214 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:20.507179 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-2gc8z_9cacdb56-27c0-4b69-ae05-8a9d2344972d/server/0.log" Oct 29 23:34:22.084820 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:22.084794 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-482f7889-lk7kd_85996d66-20b5-4d3f-b915-8172999544ae/revoke-keys/0.log" Oct 29 23:34:23.883554 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:34:23.883514 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:34:25.038505 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:25.038471 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d4decf31-v7bg5_cf7a517b-fc28-4e18-ab4c-afcb95d27a96/revoke-keys/0.log" Oct 29 23:34:26.026386 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:26.026349 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4decf31-v7bg5"] Oct 29 23:34:26.039538 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:26.039504 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4decf31-v7bg5"] Oct 29 23:34:26.169087 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:26.169060 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf7a517b-fc28-4e18-ab4c-afcb95d27a96" path="/var/lib/kubelet/pods/cf7a517b-fc28-4e18-ab4c-afcb95d27a96/volumes" Oct 29 23:34:27.739788 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:27.739755 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dd5b9556c-7gmf5_37c1f92f-3914-412b-a7aa-0cc023dce2c0/authorino/0.log" Oct 29 23:34:28.189282 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:28.189252 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-n8p4p_47662ab4-2668-46cd-9e6b-bad98897c972/manager/0.log" Oct 29 23:34:28.374073 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:28.374042 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-rvnlb_4d5895b2-39aa-489e-8c3b-f9bf15b1a58a/kuadrant-console-plugin/0.log" Oct 29 23:34:29.503305 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:29.503264 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-khfmc_28d10fb8-128f-44db-b810-48a734f838c5/manager/0.log" Oct 29 23:34:30.660788 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:30.660758 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-tf54d_b04ee4bb-9267-47f3-8ef0-d077400db137/discovery/0.log" Oct 29 23:34:30.858830 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:30.858799 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fbc7f7f7f-ggn5j_6b28ccf4-c3ec-4048-8e70-8963b25be172/kube-auth-proxy/0.log" Oct 29 23:34:31.228753 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:31.228726 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-78xhg_0e871ef2-d293-4237-b33c-996d7637d5c1/istio-proxy/0.log" Oct 29 23:34:31.600878 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:31.600849 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-djzhw_084d3c77-3d39-4234-b601-df4b9226f135/istio-proxy/0.log" Oct 29 23:34:31.786075 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:31.786039 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-lwrzr_ee04de80-4a76-4efd-8218-1f613b6130f0/istio-proxy/1.log" Oct 29 23:34:32.516058 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:32.516029 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7_2985a058-86e6-4d98-8bc5-307e78b3967c/istio-proxy/0.log" Oct 29 23:34:33.546588 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:33.546553 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-lwrzr_ee04de80-4a76-4efd-8218-1f613b6130f0/istio-proxy/0.log" Oct 29 23:34:52.888285 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:52.888255 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ldgnp_fd35650d-2c53-4230-8518-c4cb05b59558/global-pull-secret-syncer/0.log" Oct 29 23:34:53.884069 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:34:53.884028 4821 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k7c6q" containerName="extractor" Oct 29 23:34:54.028855 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:54.028824 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jzjsw_0061f2cf-c501-47b0-a89b-3e244f9c313d/konnectivity-agent/0.log" Oct 29 23:34:54.186498 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:34:54.186410 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-252.ec2.internal_83fee9c1d2770428373c9940b59a21be/haproxy/0.log" Oct 29 23:35:01.220832 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:01.220760 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6dd5b9556c-7gmf5_37c1f92f-3914-412b-a7aa-0cc023dce2c0/authorino/0.log" Oct 29 23:35:01.474937 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:01.474859 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-n8p4p_47662ab4-2668-46cd-9e6b-bad98897c972/manager/0.log" Oct 29 23:35:01.613317 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:01.613293 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-rvnlb_4d5895b2-39aa-489e-8c3b-f9bf15b1a58a/kuadrant-console-plugin/0.log" Oct 29 23:35:02.457251 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:02.457193 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-khfmc_28d10fb8-128f-44db-b810-48a734f838c5/manager/0.log" Oct 29 23:35:04.271027 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.270989 4821 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9"] Oct 29 23:35:04.271578 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271500 4821 cpu_manager.go:414] "containerMap: removing container" podUID="9a821056-9bb0-4430-ab80-d6c44d3fa4f5" containerName="payload-pre-processing" Oct 29 23:35:04.271578 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271522 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a821056-9bb0-4430-ab80-d6c44d3fa4f5" containerName="payload-pre-processing" Oct 29 23:35:04.271578 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271542 4821 cpu_manager.go:414] "containerMap: removing container" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" Oct 29 23:35:04.271578 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271549 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" Oct 29 23:35:04.271578 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271560 4821 cpu_manager.go:414] "containerMap: removing container" podUID="85996d66-20b5-4d3f-b915-8172999544ae" containerName="revoke-keys" Oct 29 23:35:04.271578 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271567 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85996d66-20b5-4d3f-b915-8172999544ae" containerName="revoke-keys" Oct 29 23:35:04.271578 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271579 4821 cpu_manager.go:414] "containerMap: removing container" podUID="ea79c88e-57ac-4f8f-a392-bf31d397cae8" containerName="istio-proxy" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271587 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea79c88e-57ac-4f8f-a392-bf31d397cae8" containerName="istio-proxy" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271598 4821 cpu_manager.go:414] "containerMap: removing container" podUID="9a799d83-b2e3-48c2-bb3a-acbffdaee0d2" containerName="maas-api" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271604 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a799d83-b2e3-48c2-bb3a-acbffdaee0d2" containerName="maas-api" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271617 4821 cpu_manager.go:414] "containerMap: removing container" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271625 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271636 4821 cpu_manager.go:414] "containerMap: removing container" podUID="9be68e75-df62-4a43-9ae9-20c04683f4c7" containerName="istio-proxy" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271643 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9be68e75-df62-4a43-9ae9-20c04683f4c7" containerName="istio-proxy" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271661 4821 cpu_manager.go:414] "containerMap: removing container" podUID="bab23b35-8838-415c-90bf-aebf41448b58" containerName="maas-api" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271668 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab23b35-8838-415c-90bf-aebf41448b58" containerName="maas-api" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271681 4821 cpu_manager.go:414] "containerMap: removing container" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271688 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271698 4821 cpu_manager.go:414] "containerMap: removing container" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271705 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271717 4821 cpu_manager.go:414] "containerMap: removing container" podUID="f3f2d74f-e211-41ce-8ed6-432892f64365" containerName="istio-proxy" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271724 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f2d74f-e211-41ce-8ed6-432892f64365" containerName="istio-proxy" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271735 4821 cpu_manager.go:414] "containerMap: removing container" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271743 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271756 4821 cpu_manager.go:414] "containerMap: removing container" podUID="9809020b-a844-44fa-86af-6565434de12d" containerName="payload-processing" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271764 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9809020b-a844-44fa-86af-6565434de12d" containerName="payload-processing" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271776 4821 cpu_manager.go:414] "containerMap: removing container" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271784 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271803 4821 cpu_manager.go:414] "containerMap: removing container" podUID="f3f2d74f-e211-41ce-8ed6-432892f64365" containerName="istio-proxy" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271811 4821 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f2d74f-e211-41ce-8ed6-432892f64365" containerName="istio-proxy" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271890 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab23b35-8838-415c-90bf-aebf41448b58" containerName="maas-api" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271908 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="c436ecf5-3fdf-43ea-be7e-d58eaf7481b6" containerName="main" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271917 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="59c07a4a-dee9-4e00-a10b-eee2f9160fd1" containerName="main" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271926 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f2d74f-e211-41ce-8ed6-432892f64365" containerName="istio-proxy" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271936 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e04ee6b-8e5b-4dc1-b001-184689f651e5" containerName="main" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271946 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="9809020b-a844-44fa-86af-6565434de12d" containerName="payload-processing" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271955 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f5dfd6-1e9c-447d-ac89-82393448335d" containerName="main" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271965 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a821056-9bb0-4430-ab80-d6c44d3fa4f5" containerName="payload-pre-processing" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271974 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeacdd0d-a01d-498c-bf12-f384dc79d57f" containerName="main" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271982 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="85996d66-20b5-4d3f-b915-8172999544ae" containerName="revoke-keys" Oct 29 23:35:04.271977 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271991 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea79c88e-57ac-4f8f-a392-bf31d397cae8" containerName="istio-proxy" Oct 29 23:35:04.273532 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.271999 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a799d83-b2e3-48c2-bb3a-acbffdaee0d2" containerName="maas-api" Oct 29 23:35:04.273532 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.272008 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d3f4d01-15ca-4d75-a69b-a5adff178017" containerName="main" Oct 29 23:35:04.273532 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.272016 4821 memory_manager.go:365] "RemoveStaleState removing state" podUID="9be68e75-df62-4a43-9ae9-20c04683f4c7" containerName="istio-proxy" Oct 29 23:35:04.275430 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.275402 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9" Oct 29 23:35:04.296057 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.296035 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9"] Oct 29 23:35:04.307036 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.307008 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zfxq6\"/\"openshift-service-ca.crt\"" Oct 29 23:35:04.307242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.307008 4821 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zfxq6\"/\"default-dockercfg-gxv8f\"" Oct 29 23:35:04.307242 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.307057 4821 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zfxq6\"/\"kube-root-ca.crt\"" Oct 29 23:35:04.341821 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.341791 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw"] Oct 29 23:35:04.342010 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.341990 4821 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" podUID="084d3c77-3d39-4234-b601-df4b9226f135" containerName="istio-proxy" containerID="cri-o://da8f801a56cb07cb1063ef06fd06d4a7c0b1e584e814c639d72cb374c82e8586" gracePeriod=30 Oct 29 23:35:04.419889 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.419850 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/76358c28-77c8-470e-97a4-987431c0b7db-podres\") pod \"perf-node-gather-daemonset-d9dm9\" (UID: \"76358c28-77c8-470e-97a4-987431c0b7db\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9" Oct 29 23:35:04.419889 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.419881 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76358c28-77c8-470e-97a4-987431c0b7db-sys\") pod \"perf-node-gather-daemonset-d9dm9\" (UID: \"76358c28-77c8-470e-97a4-987431c0b7db\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9" Oct 29 23:35:04.420104 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.419964 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76358c28-77c8-470e-97a4-987431c0b7db-lib-modules\") pod \"perf-node-gather-daemonset-d9dm9\" (UID: \"76358c28-77c8-470e-97a4-987431c0b7db\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9" Oct 29 23:35:04.420104 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.419991 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/76358c28-77c8-470e-97a4-987431c0b7db-proc\") pod \"perf-node-gather-daemonset-d9dm9\" (UID: \"76358c28-77c8-470e-97a4-987431c0b7db\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9" Oct 29 23:35:04.420104 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.420020 4821 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h45nc\" (UniqueName: \"kubernetes.io/projected/76358c28-77c8-470e-97a4-987431c0b7db-kube-api-access-h45nc\") pod \"perf-node-gather-daemonset-d9dm9\" (UID: \"76358c28-77c8-470e-97a4-987431c0b7db\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9" Oct 29 23:35:04.521345 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.521258 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76358c28-77c8-470e-97a4-987431c0b7db-lib-modules\") pod \"perf-node-gather-daemonset-d9dm9\" (UID: \"76358c28-77c8-470e-97a4-987431c0b7db\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9" Oct 29 23:35:04.521345 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.521296 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/76358c28-77c8-470e-97a4-987431c0b7db-proc\") pod \"perf-node-gather-daemonset-d9dm9\" (UID: \"76358c28-77c8-470e-97a4-987431c0b7db\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9" Oct 29 23:35:04.521552 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.521371 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/76358c28-77c8-470e-97a4-987431c0b7db-proc\") pod \"perf-node-gather-daemonset-d9dm9\" (UID: \"76358c28-77c8-470e-97a4-987431c0b7db\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9" Oct 29 23:35:04.521552 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.521395 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76358c28-77c8-470e-97a4-987431c0b7db-lib-modules\") pod \"perf-node-gather-daemonset-d9dm9\" (UID: \"76358c28-77c8-470e-97a4-987431c0b7db\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9" Oct 29 23:35:04.521552 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.521426 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h45nc\" (UniqueName: \"kubernetes.io/projected/76358c28-77c8-470e-97a4-987431c0b7db-kube-api-access-h45nc\") pod \"perf-node-gather-daemonset-d9dm9\" (UID: \"76358c28-77c8-470e-97a4-987431c0b7db\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9" Oct 29 23:35:04.521552 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.521513 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/76358c28-77c8-470e-97a4-987431c0b7db-podres\") pod \"perf-node-gather-daemonset-d9dm9\" (UID: \"76358c28-77c8-470e-97a4-987431c0b7db\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9" Oct 29 23:35:04.521552 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.521537 4821 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76358c28-77c8-470e-97a4-987431c0b7db-sys\") pod \"perf-node-gather-daemonset-d9dm9\" (UID: \"76358c28-77c8-470e-97a4-987431c0b7db\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9" Oct 29 23:35:04.521739 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.521605 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/76358c28-77c8-470e-97a4-987431c0b7db-podres\") pod \"perf-node-gather-daemonset-d9dm9\" (UID: \"76358c28-77c8-470e-97a4-987431c0b7db\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9" Oct 29 23:35:04.521739 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.521642 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76358c28-77c8-470e-97a4-987431c0b7db-sys\") pod \"perf-node-gather-daemonset-d9dm9\" (UID: \"76358c28-77c8-470e-97a4-987431c0b7db\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9" Oct 29 23:35:04.545853 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.545824 4821 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h45nc\" (UniqueName: \"kubernetes.io/projected/76358c28-77c8-470e-97a4-987431c0b7db-kube-api-access-h45nc\") pod \"perf-node-gather-daemonset-d9dm9\" (UID: \"76358c28-77c8-470e-97a4-987431c0b7db\") " pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9" Oct 29 23:35:04.587573 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.587547 4821 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9" Oct 29 23:35:04.671003 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.670972 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-j26l4_8d65af4a-3d99-4d72-94ce-cd7d5aee4182/manager/0.log" Oct 29 23:35:04.799157 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.799085 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-j26l4_8d65af4a-3d99-4d72-94ce-cd7d5aee4182/manager/0.log" Oct 29 23:35:04.892001 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:04.891966 4821 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9"] Oct 29 23:35:04.895130 ip-10-0-130-252 kubenswrapper[4821]: W1029 23:35:04.895103 4821 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod76358c28_77c8_470e_97a4_987431c0b7db.slice/crio-df55866ebe2d431fd3313a77ea4642f7801fb5f49286cb86aac2ccebd8c3d9d8 WatchSource:0}: Error finding container df55866ebe2d431fd3313a77ea4642f7801fb5f49286cb86aac2ccebd8c3d9d8: Status 404 returned error can't find the container with id df55866ebe2d431fd3313a77ea4642f7801fb5f49286cb86aac2ccebd8c3d9d8 Oct 29 23:35:05.384984 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:05.384934 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9" event={"ID":"76358c28-77c8-470e-97a4-987431c0b7db","Type":"ContainerStarted","Data":"0e632694b68ab520f019ecc71ab4212cfd9dc5c307ce2ec0340f5f22875d66a9"} Oct 29 23:35:05.385536 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:05.385034 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9" event={"ID":"76358c28-77c8-470e-97a4-987431c0b7db","Type":"ContainerStarted","Data":"df55866ebe2d431fd3313a77ea4642f7801fb5f49286cb86aac2ccebd8c3d9d8"} Oct 29 23:35:05.385536 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:05.385078 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9" Oct 29 23:35:05.780967 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:05.780879 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rx2tt_e781f3f8-7c1e-4f60-b8a7-236e97928c29/manager/0.log" Oct 29 23:35:05.919259 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:05.919222 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rx2tt_e781f3f8-7c1e-4f60-b8a7-236e97928c29/manager/0.log" Oct 29 23:35:06.165922 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:06.165889 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-2gc8z_9cacdb56-27c0-4b69-ae05-8a9d2344972d/server/0.log" Oct 29 23:35:06.305983 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:06.305953 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-shnd6_4916614f-8a34-491d-b4ad-bae72c14b140/node-exporter/0.log" Oct 29 23:35:06.318503 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:06.318476 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-2gc8z_9cacdb56-27c0-4b69-ae05-8a9d2344972d/server/0.log" Oct 29 23:35:06.432755 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:06.432685 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-shnd6_4916614f-8a34-491d-b4ad-bae72c14b140/kube-rbac-proxy/0.log" Oct 29 23:35:06.560167 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:06.560139 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-shnd6_4916614f-8a34-491d-b4ad-bae72c14b140/init-textfile/0.log" Oct 29 23:35:09.446274 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.446250 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-j26l4_8d65af4a-3d99-4d72-94ce-cd7d5aee4182/manager/0.log" Oct 29 23:35:09.592966 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.592942 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:35:09.642276 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.642160 4821 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9" podStartSLOduration=5.642141884 podStartE2EDuration="5.642141884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:35:05.428268025 +0000 UTC m=+3159.368815934" watchObservedRunningTime="2026-10-29 23:35:09.642141884 +0000 UTC m=+3163.582689794" Oct 29 23:35:09.668426 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.668385 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-istio-data\") pod \"084d3c77-3d39-4234-b601-df4b9226f135\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " Oct 29 23:35:09.668671 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.668451 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/084d3c77-3d39-4234-b601-df4b9226f135-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/084d3c77-3d39-4234-b601-df4b9226f135-istio-podinfo\") pod \"084d3c77-3d39-4234-b601-df4b9226f135\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " Oct 29 23:35:09.668671 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.668474 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-istio-envoy\") pod \"084d3c77-3d39-4234-b601-df4b9226f135\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " Oct 29 23:35:09.668671 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.668514 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-workload-socket\") pod \"084d3c77-3d39-4234-b601-df4b9226f135\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " Oct 29 23:35:09.668671 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.668541 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-credential-socket\") pod \"084d3c77-3d39-4234-b601-df4b9226f135\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " Oct 29 23:35:09.668671 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.668600 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/084d3c77-3d39-4234-b601-df4b9226f135-istio-token\" (UniqueName: \"kubernetes.io/projected/084d3c77-3d39-4234-b601-df4b9226f135-istio-token\") pod \"084d3c77-3d39-4234-b601-df4b9226f135\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " Oct 29 23:35:09.668671 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.668640 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/084d3c77-3d39-4234-b601-df4b9226f135-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/084d3c77-3d39-4234-b601-df4b9226f135-istiod-ca-cert\") pod \"084d3c77-3d39-4234-b601-df4b9226f135\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " Oct 29 23:35:09.668671 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.668660 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-workload-certs\") pod \"084d3c77-3d39-4234-b601-df4b9226f135\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " Oct 29 23:35:09.669244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.668701 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-istio-data" pod "084d3c77-3d39-4234-b601-df4b9226f135" (UID: "084d3c77-3d39-4234-b601-df4b9226f135"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:35:09.669244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.668731 4821 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/084d3c77-3d39-4234-b601-df4b9226f135-kube-api-access-qsq7t\" (UniqueName: \"kubernetes.io/projected/084d3c77-3d39-4234-b601-df4b9226f135-kube-api-access-qsq7t\") pod \"084d3c77-3d39-4234-b601-df4b9226f135\" (UID: \"084d3c77-3d39-4234-b601-df4b9226f135\") " Oct 29 23:35:09.669244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.668991 4821 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-istio-data\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:35:09.669244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.669118 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-credential-socket" pod "084d3c77-3d39-4234-b601-df4b9226f135" (UID: "084d3c77-3d39-4234-b601-df4b9226f135"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:35:09.669244 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.669152 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/084d3c77-3d39-4234-b601-df4b9226f135-istiod-ca-cert" pod "084d3c77-3d39-4234-b601-df4b9226f135" (UID: "084d3c77-3d39-4234-b601-df4b9226f135"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:35:09.669577 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.669386 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-workload-certs" pod "084d3c77-3d39-4234-b601-df4b9226f135" (UID: "084d3c77-3d39-4234-b601-df4b9226f135"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:35:09.669577 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.669499 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-workload-socket" pod "084d3c77-3d39-4234-b601-df4b9226f135" (UID: "084d3c77-3d39-4234-b601-df4b9226f135"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:35:09.671301 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.671274 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/084d3c77-3d39-4234-b601-df4b9226f135-istio-podinfo" pod "084d3c77-3d39-4234-b601-df4b9226f135" (UID: "084d3c77-3d39-4234-b601-df4b9226f135"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:35:09.671480 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.671315 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084d3c77-3d39-4234-b601-df4b9226f135-istio-token" pod "084d3c77-3d39-4234-b601-df4b9226f135" (UID: "084d3c77-3d39-4234-b601-df4b9226f135"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:35:09.671742 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.671721 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084d3c77-3d39-4234-b601-df4b9226f135-kube-api-access-qsq7t" pod "084d3c77-3d39-4234-b601-df4b9226f135" (UID: "084d3c77-3d39-4234-b601-df4b9226f135"). InnerVolumeSpecName "kube-api-access-qsq7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:35:09.671837 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.671813 4821 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-istio-envoy" pod "084d3c77-3d39-4234-b601-df4b9226f135" (UID: "084d3c77-3d39-4234-b601-df4b9226f135"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:35:09.693780 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.693759 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-j26l4_8d65af4a-3d99-4d72-94ce-cd7d5aee4182/manager/0.log" Oct 29 23:35:09.769458 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.769428 4821 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/084d3c77-3d39-4234-b601-df4b9226f135-istio-podinfo\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:35:09.769458 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.769453 4821 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-istio-envoy\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:35:09.769458 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.769463 4821 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-workload-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:35:09.769689 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.769475 4821 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-credential-socket\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:35:09.769689 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.769490 4821 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/084d3c77-3d39-4234-b601-df4b9226f135-istio-token\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:35:09.769689 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.769501 4821 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/084d3c77-3d39-4234-b601-df4b9226f135-istiod-ca-cert\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:35:09.769689 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.769513 4821 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/084d3c77-3d39-4234-b601-df4b9226f135-workload-certs\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:35:09.769689 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:09.769526 4821 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsq7t\" (UniqueName: \"kubernetes.io/projected/084d3c77-3d39-4234-b601-df4b9226f135-kube-api-access-qsq7t\") on node \"ip-10-0-130-252.ec2.internal\" DevicePath \"\"" Oct 29 23:35:10.413108 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:10.413066 4821 generic.go:356] "Generic (PLEG): container finished" podID="084d3c77-3d39-4234-b601-df4b9226f135" containerID="da8f801a56cb07cb1063ef06fd06d4a7c0b1e584e814c639d72cb374c82e8586" exitCode=0 Oct 29 23:35:10.413342 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:10.413106 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" event={"ID":"084d3c77-3d39-4234-b601-df4b9226f135","Type":"ContainerDied","Data":"da8f801a56cb07cb1063ef06fd06d4a7c0b1e584e814c639d72cb374c82e8586"} Oct 29 23:35:10.413342 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:10.413149 4821 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" Oct 29 23:35:10.413342 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:10.413220 4821 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw" event={"ID":"084d3c77-3d39-4234-b601-df4b9226f135","Type":"ContainerDied","Data":"d7ab1a727d5f71e71364f97ce11dec953a1960842aab0a63117e010b0e49f20d"} Oct 29 23:35:10.413342 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:10.413258 4821 scope.go:122] "RemoveContainer" containerID="da8f801a56cb07cb1063ef06fd06d4a7c0b1e584e814c639d72cb374c82e8586" Oct 29 23:35:10.425649 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:10.425589 4821 scope.go:122] "RemoveContainer" containerID="da8f801a56cb07cb1063ef06fd06d4a7c0b1e584e814c639d72cb374c82e8586" Oct 29 23:35:10.425988 ip-10-0-130-252 kubenswrapper[4821]: E1029 23:35:10.425964 4821 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da8f801a56cb07cb1063ef06fd06d4a7c0b1e584e814c639d72cb374c82e8586\": container with ID starting with da8f801a56cb07cb1063ef06fd06d4a7c0b1e584e814c639d72cb374c82e8586 not found: ID does not exist" containerID="da8f801a56cb07cb1063ef06fd06d4a7c0b1e584e814c639d72cb374c82e8586" Oct 29 23:35:10.426053 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:10.426000 4821 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da8f801a56cb07cb1063ef06fd06d4a7c0b1e584e814c639d72cb374c82e8586"} err="failed to get container status \"da8f801a56cb07cb1063ef06fd06d4a7c0b1e584e814c639d72cb374c82e8586\": rpc error: code = NotFound desc = could not find container \"da8f801a56cb07cb1063ef06fd06d4a7c0b1e584e814c639d72cb374c82e8586\": container with ID starting with da8f801a56cb07cb1063ef06fd06d4a7c0b1e584e814c639d72cb374c82e8586 not found: ID does not exist" Oct 29 23:35:10.489842 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:10.489768 4821 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw"] Oct 29 23:35:10.502442 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:10.502413 4821 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-djzhw"] Oct 29 23:35:10.513257 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:10.513232 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rx2tt_e781f3f8-7c1e-4f60-b8a7-236e97928c29/manager/0.log" Oct 29 23:35:10.772648 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:10.772514 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rx2tt_e781f3f8-7c1e-4f60-b8a7-236e97928c29/manager/0.log" Oct 29 23:35:10.883133 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:10.883102 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-2gc8z_9cacdb56-27c0-4b69-ae05-8a9d2344972d/server/0.log" Oct 29 23:35:11.155467 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:11.155416 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-2gc8z_9cacdb56-27c0-4b69-ae05-8a9d2344972d/server/0.log" Oct 29 23:35:11.401852 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:11.401823 4821 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zfxq6/perf-node-gather-daemonset-d9dm9" Oct 29 23:35:12.168797 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:12.168766 4821 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="084d3c77-3d39-4234-b601-df4b9226f135" path="/var/lib/kubelet/pods/084d3c77-3d39-4234-b601-df4b9226f135/volumes" Oct 29 23:35:14.679424 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:14.679383 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-j26l4_8d65af4a-3d99-4d72-94ce-cd7d5aee4182/manager/0.log" Oct 29 23:35:14.814792 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:14.814762 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-j26l4_8d65af4a-3d99-4d72-94ce-cd7d5aee4182/manager/0.log" Oct 29 23:35:15.759569 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:15.759517 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rx2tt_e781f3f8-7c1e-4f60-b8a7-236e97928c29/manager/0.log" Oct 29 23:35:15.988047 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:15.988009 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rx2tt_e781f3f8-7c1e-4f60-b8a7-236e97928c29/manager/0.log" Oct 29 23:35:16.097596 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:16.097517 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-2gc8z_9cacdb56-27c0-4b69-ae05-8a9d2344972d/server/0.log" Oct 29 23:35:17.120769 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:17.120740 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-2gc8z_9cacdb56-27c0-4b69-ae05-8a9d2344972d/server/0.log" Oct 29 23:35:18.874058 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:18.874024 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zb2ch_1ae4b768-8a6b-4c10-ae02-c1a3950d830c/dns/0.log" Oct 29 23:35:19.000548 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:19.000509 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zb2ch_1ae4b768-8a6b-4c10-ae02-c1a3950d830c/kube-rbac-proxy/0.log" Oct 29 23:35:19.132517 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:19.132439 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cw2sv_33360cd4-4738-4c3d-bc9f-9f5003408589/dns-node-resolver/0.log" Oct 29 23:35:21.037052 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:21.037012 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7kgv6_f99954e5-6dfe-457f-9a8f-22752798bb5d/node-ca/0.log" Oct 29 23:35:22.140630 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:22.140594 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-tf54d_b04ee4bb-9267-47f3-8ef0-d077400db137/discovery/0.log" Oct 29 23:35:22.174934 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:22.174907 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fbc7f7f7f-ggn5j_6b28ccf4-c3ec-4048-8e70-8963b25be172/kube-auth-proxy/0.log" Oct 29 23:35:22.297767 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:22.297732 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-78xhg_0e871ef2-d293-4237-b33c-996d7637d5c1/istio-proxy/0.log" Oct 29 23:35:22.431181 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:22.431095 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-lwrzr_ee04de80-4a76-4efd-8218-1f613b6130f0/istio-proxy/0.log" Oct 29 23:35:22.431572 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:22.431552 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-lwrzr_ee04de80-4a76-4efd-8218-1f613b6130f0/istio-proxy/1.log" Oct 29 23:35:22.645474 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:22.645443 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-69694c4c9-n44k7_2985a058-86e6-4d98-8bc5-307e78b3967c/istio-proxy/0.log" Oct 29 23:35:23.179525 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:23.179484 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-tf54d_b04ee4bb-9267-47f3-8ef0-d077400db137/discovery/0.log" Oct 29 23:35:23.219322 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:23.219296 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5fbc7f7f7f-ggn5j_6b28ccf4-c3ec-4048-8e70-8963b25be172/kube-auth-proxy/0.log" Oct 29 23:35:23.349298 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:23.349267 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-78xhg_0e871ef2-d293-4237-b33c-996d7637d5c1/istio-proxy/0.log" Oct 29 23:35:23.490353 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:23.490259 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-lwrzr_ee04de80-4a76-4efd-8218-1f613b6130f0/istio-proxy/0.log" Oct 29 23:35:23.490863 ip-10-0-130-252 kubenswrapper[4821]: I1029 23:35:23.490836 4821 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-lwrzr_ee04de80-4a76-4efd-8218-1f613b6130f0/istio-proxy/1.log"